Augmented reality interaction techniques

ABSTRACT

A method may include receiving, via a processor, image data associated with a user&#39;s surrounding and generating, via the processor, a visualization that may include a virtual industrial automation device. The virtual industrial automation device may depict a virtual object within image data, and the virtual object may correspond to a physical industrial automation device. The method may include displaying, via the processor, the visualization via an electronic display and detecting, via the processor, a gesture in image data that may include the user&#39;s surrounding and the visualization. The gesture may be indicative of a request to move the virtual industrial automation device. The method may include tracking, via the processor, a user&#39;s movement, generating, via the processor, a visualization that may include an animation of the virtual industrial automation device moving based on the user&#39;s movement, and displaying, via the processor, the visualization via the electronic display.

BACKGROUND

The disclosure relates generally to the design of industrial systems.More particularly, embodiments of the present disclosure are related tosystems and methods for detecting user input within an augmented realityenvironment and displaying or modifying visualizations associated withan industrial automation device or an industrial system based on thedetected user input.

Augmented reality (AR) devices provide layers of computer-generatedcontent superimposed on a visualization of a real-world environment to auser via a display. That is, an AR environment may provide a user with acombination of real-world content and computer-generated content.Augmented reality devices may include, for example, a head mounteddevice, smart glasses, a virtual retinal display, a contact lens, acomputer, or a hand-held device, such as a mobile phone or a tablet. AsAR devices become more widely available, these devices may be used toassist operators in industrial automation environments to performcertain tasks. As such, it is recognized that improved systems andmethods for perform certain operations in the AR environment may betterenable the operators to perform their job functions for efficiently.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present techniques,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

BRIEF DESCRIPTION

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

In one embodiment, a system for interacting with virtual objects in anaugmented reality environment may include a head mounted device. Thehead mounted device may receive a first set of image data associatedwith a surrounding of a user and generate a first visualizationcomprising a plurality of virtual compartments. Each virtual compartmentmay be associated with one type of virtual industrial automation deviceand each virtual compartment may include a plurality of virtualindustrial automation devices. Each virtual industrial automation devicemay depict a virtual object within the first set of image data and thevirtual object may correspond to a physical industrial automationdevice. The head mounted device may display the first visualization viaan electronic display and detect a gesture in a second set of image datathat may include the surrounding of the user and the firstvisualization. The gesture may be indicative of a selection of one ofthe plurality of virtual compartments. The head mounted device maygenerate a second visualization comprising a respective plurality ofvirtual industrial automation devices associated with the selection anddisplay the second visualization via the electronic display.

In another embodiment, a method may include receiving, via a processor,a first set of image data associated with a surrounding of a user andgenerating, via the processor, a first visualization comprising avirtual industrial automation device. The virtual industrial automationdevice may be configured to depict a virtual object within the first setof image data and the virtual object may correspond to a physicalindustrial automation device. The method may include displaying, via theprocessor, the first visualization via an electronic display anddetecting, via the processor, a gesture in a second set of image datathat may include the surrounding of the user and the firstvisualization. The gesture may be indicative of a request to move thevirtual industrial automation device. The method may include tracking,via the processor, a movement of the user, generating, via theprocessor, a second visualization that may include an animation of thevirtual industrial automation device moving based on the movement, anddisplaying, via the processor, the second visualization via theelectronic display.

In yet another embodiment, a computer-readable medium may includecomputer-executable instructions that, when executed, may cause aprocessor to receive a first set of image data associated with asurrounding of a user and generate a first visualization that mayinclude a first virtual industrial automation device and a secondvirtual industrial automation device. The first and second virtualindustrial automation devices may depict first and second respectivevirtual objects within the first set of image data, and the first andsecond respective virtual objects may correspond to a first and a secondphysical industrial automation device. The computer-readable medium mayinclude computer-executable instructions that, when executed, may causethe processor to display the first visualization via an electronicdisplay and detect a first gesture in a second set of image data thatmay include the surrounding of the user and the first visualization. Thefirst gesture may be indicative of a movement of the first virtualindustrial automation device toward the second virtual industrialautomation device. The computer-readable medium may includecomputer-executable instructions that, when executed, may cause theprocessor to determine a compatibility between the first virtualindustrial automation device and the second virtual industrialautomation device, generate a second visualization that may include ananimation of the first virtual industrial automation device coupling tothe second virtual industrial automation device to create a jointvirtual industrial automation device in response to determining that thefirst virtual industrial automation device and the second virtualindustrial automation device are compatible, generate a thirdvisualization comprising an error notification in response todetermining that the first virtual industrial automation device and thesecond virtual industrial automation device are incompatible, anddisplay the second visualization or the third visualization via theelectronic display.

DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure may become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a block diagram of an exemplary embodiment of an interactiveaugmented reality (AR) system that may be utilized to display andinteract with a virtual representation of an industrial automationsystem in an AR environment, in accordance with an embodiment;

FIG. 2 a block diagram of an exemplary display device of the interactiveAR system of FIG. 1, in accordance with an embodiment;

FIG. 3 is a perspective view of an exemplary visualization that may beperceived by a user utilizing the display device of FIG. 2 before theperformance of a first gaze gesture command, in accordance with anembodiment;

FIG. 4 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 after aperformance of the first gaze gesture command perform a second gazegesture command, in accordance with an embodiment;

FIG. 5 is perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming the second gaze gesture command, in accordance with anembodiment;

FIG. 6 is a flowchart of a method for displaying and modifying avisualization based on one or more gesture commands using the displaydevice of FIG. 2, in accordance with an embodiment;

FIG. 7 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming a gaze gesture command, in accordance with an embodiment;

FIG. 8 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 beforeperforming a grasp gesture command, in accordance with an embodiment;

FIG. 9 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming a grasp gesture command, in accordance with an embodiment;

FIG. 10 is a flowchart of a method for displaying and modifying avisualization based on a grasp gesture command using the display deviceof FIG. 2, in accordance with an embodiment.

FIG. 11 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 beforeperforming a push gesture command, in accordance with an embodiment;

FIG. 12 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming a push gesture command, in accordance with an embodiment;

FIG. 13 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 beforeperforming a rotate gesture command, in accordance with an embodiment;

FIG. 14 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming a rotate gesture command, in accordance with an embodiment;

FIG. 15 is a flowchart of a method for displaying and modifying avisualization based on a push gesture command, a pull gesture command,or a rotate gesture command using the display device of FIG. 2, inaccordance with an embodiment.

FIG. 16 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 beforeperforming a snap gesture command, in accordance with an embodiment;

FIG. 17 is a perspective view of an exemplary visualization that may beperceived by the user utilizing the display device of FIG. 2 afterperforming a snap gesture command, in accordance with an embodiment;

FIG. 18 is a flowchart of a method for displaying and modifying avisualization based on a snap gesture command or a separate gesturecommand using the display device of FIG. 2, in accordance with anembodiment.

FIG. 19 is a perspective view of an exemplary visualization that may beperceived by a user utilizing the display device of FIG. 2 in a dynamicrotation mode, in accordance with an embodiment; and

FIG. 20 is a perspective view of an exemplary visualization that may beperceived by a user utilizing the display device of FIG. 2 afterperforming a scale down command, in accordance with an embodiment;

DETAILED DESCRIPTION

One or more specific embodiments of the present disclosure will bedescribed below. In an effort to provide a concise description of theseembodiments, all features of an actual implementation may not bedescribed in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements. One ormore specific embodiments of the present embodiments described hereinwill be described below. In an effort to provide a concise descriptionof these embodiments, all features of an actual implementation may notbe described in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

The present disclosure is generally directed towards an interactiveaugmented reality (AR) system that may display one or morevisualizations of a combination of real-world and computer-generatedcontent in an AR environment to a user and detect various gesturesperformed by the user that correspond to respective interactions withthe real-world and computer-generated content in the AR environment.After detecting a gesture performed by the user, the interactive ARsystem may modify a visualization associated with the AR environmentbased on the detected gesture. For example, the interactive AR systemmay alter the user's perception of the real world by displaying ormodifying one or more virtual objects in the visualization associatedwith the AR environment presented to the user based on the detectedgesture. In some embodiments, modifying a visualization or modifying avirtual object in a visualization may include generating an additionalvisualization and displaying the additional visualization.

Although the AR system is described herein as providingcomputer-generated content visually, it should be noted that the ARsystem may provide computer-generated content via other types of sensorymodalities. For example, the computer-generated content may be presentedto a user via an auditory modality, a haptic modality, a somatosensorymodality, an olfactory modality, or the like. Additionally, although theinteractive AR system is described herein as providing an AR experienceto the user, it should be noted that features of the interactive ARsystem may be utilized within a virtual reality context or a mixedreality context as well. In one embodiment, the interactive AR systemmay generate and display a visualization that contains real-worldcontent (e.g., the user's surroundings or portions of the user) andcomputer-generated content (e.g., virtual objects). In anotherembodiment, the interactive AR system may have a transparent display anddisplay a visualization of computer-generated content (e.g., virtualobjects) superimposed over the transparent display to produce virtualobjects within real world surroundings.

In certain embodiments, the interactive AR system may also detectvarious voice commands that correspond to respective interactions withthe computer-generated content in the AR environment. After detecting avoice command generated by the user, the interactive AR system maymodify the AR environment based on the detected voice command. In someembodiments, a voice command may correspond to an interaction similar toan interaction associated with a gesture. In this way, the interactiveAR system may provide the user with flexible control of the interactiveAR system by facilitating the user's control of the interactive ARsystem in different ways.

For ease of discussion, the description of the interactive AR systemprovided herein is made with reference to the design of an industrialsystem. However, it should be noted that the interactive AR system, asdescribed herein, is not limited to such embodiments. The interactive ARsystem may be used in various other fields and applications. Forexample, the interactive AR system may be used to formulate possiblearchaeological site configurations, visualize and design constructionmodels or sites of residential or commercial buildings, undergroundstructures, or offshore wells, visualize and design commercial productsor previews of commercial products, such as furniture, clothing,appliances, or vehicles, provide educational material or training,visualize local or remote geographic features, such as cities, forests,or oceans, facilitate social interaction, provide digital game play in areal world environment, provide medical information, such as X-ray,ultrasound, or magnetic resonance imaging (MRI) data, facilitate orvisualize toy manufacturing or toy design, or the like.

For an industrial system with many components, it may be beneficial totailor the design of the industrial system to the real-world environmentin which the industrial system may be located after assembly of theindustrial system. For example, in an industrial system employingconveyors, such as a high-speed packaging line, the design of theindustrial system may be constrained by the physical dimensions or theshape of the building in which the industrial system may be located.Additionally, each conveyor in the industrial system may have a plethoraof parts, such as conveyor sections, movers, and the like, that eachhaving a different size and shape. Further, the design of the industrialsystem may be tailored to the needs of a user of the industrial system.For example, the specific configuration and shape of the conveyors ofthe industrial system may be based on other components within theindustrial system that provide product to the conveyor to transport. Assuch, it may be beneficial to visualize and design an industrial systemwithin the real-world environment that the industrial system may belocated before assembly of the industrial system to optimize the designof the industrial system based on the physical constraints of thelocation and the needs of the user of the industrial system.

With the foregoing in mind, the interactive AR system may facilitate thevisualization and the design of an industrial system by a user withinthe physical space that the industrial system may be located afterassembly. For example, the interactive AR system may display avisualization associated with various virtual industrial automationdevices, such as conveyor sections, movers, or the like, to the user inan AR environment, while the user is present in the real-worldenvironment that the actual industrial automation devices may be locatedafter assembly. The interactive AR system may then detect a gestureperformed by the user or a voice command issued by the user and modifythe visualization based on the detected gesture or the detected voicecommand. As such, the user may visualize and model variousconfigurations and designs of an industrial system and the components ofthe industrial system within the physical space the industrial systemmay be located after assembly.

It should be noted that the gestures and voice commands provided by theuser to control the positioning of the virtual objects in the ARenvironment may be more beneficial for the user if they mimic real worldmotions. That is, using gestures and movements to move a virtual objectin the same manner a person may wish to move a real object may providefor a more desirable interface between the user and the AR environment.For example, the following gestures may be performed by the user tointeract with one or more virtual industrial automation devices of anindustrial system in an AR environment after the gestures are detectedby the interactive AR system. It should be noted that the followinggestures are exemplary and non-limiting and that other gestures may beperformed that provide similar interactions with the virtualrepresentations as described herein or similar gestures may be performedthat provide different interactions with the virtual representations asdescribed herein.

In one embodiment, the interactive AR system may detect a gaze gesture(e.g., viewing an object for certain amount of time) performed by theuser to indicate a selection of a virtual industrial automation deviceor a request for the interactive AR system to display additionalinformation associated with the industrial automation device. Forexample, after the interactive AR system has detected a gaze gestureperformed by the user and directed at a virtual industrial automationdevice, the interactive AR system may modify a visualization associatedwith the AR environment to indicate that the virtual industrialautomation device has been selected by a user. Such an indication may berepresented by a coloring, a shading, a transition of between dottedlines to solid lines, a highlighting, or the like, of the virtualindustrial automation device in the visualization associated with the ARenvironment. After the virtual industrial automation device has beenselected by the user in the visualization associated with the ARenvironment, the user may perform additional gestures as describedherein to interact with the virtual industrial automation devicefurther.

In another example, in a design environment, after the interactive ARsystem has detected a gaze gesture performed by the user and directed ata virtual compartment or category, the interactive AR system may modifya visualization associated with the AR environment to display thecontents of the virtual compartment (e.g., one or more virtualindustrial automation devices that are associated with the virtualcompartment). For example, the virtual compartment may include varioustypes of virtual conveyor sections that may be employed within thedesign of a conveyor system. After the interactive AR system hasdetected a gaze feature performed by the user and directed at thevirtual compartment, the interactive AR system may modify avisualization associated with the AR environment to display the varioustypes of virtual conveyor sections (e.g., virtual conveyor sectionshaving different shapes, such as a straight section, a U-shaped section,a C-shaped section, or the like).

In another example, after the interactive AR system has detected a gazegesture performed by the user and directed at a virtual industrialautomation device, the interactive AR system may modify a visualizationassociated with the AR environment to display various types of dataassociated with the virtual industrial automation device, such asidentification data, compatibility data with other virtual industrialautomation devices, or the like.

In another embodiment, after the user has selected a virtual industrialautomation device, the interactive AR system may detect a graspinggesture performed by the user to map the virtual industrial automationdevice to the hand of the user that performed the grasping motion. Insome embodiments, the interactive AR system may map two respectivevirtual industrial automation devices to each hand of the user. That is,the interactive AR system may detect a first grasping gesture performedby a user's first hand and map a first virtual industrial automationdevice to the user's first hand, and the interactive AR system maydetect a second grasping gesture performed by the user's second hand andmap a second virtual industrial automation device to the user's secondhand. After a virtual industrial automation device is mapped to a user'shand, the user may move the virtual industrial automation device withinthe visualization associated with the AR environment in real-time orapproximately in real-time. That is, the interactive AR system may trackthe movement of the user's hand in the visualization in the ARenvironment and modify the visualization in response to the movement ofthe user's hand such that the virtual industrial automation devicemapped to user's hand appears in the same location as the user's hand inthe visualization.

In yet another embodiment, after the interactive AR system has mapped avirtual industrial automation device to each of the user's hands, theinteractive AR system may detect a snap gesture performed by the userthat involves the user bringing both of the user's hands together. Forexample, the snap gesture may involve the user bringing both of theuser's hands together while the user is grasping a respective virtualindustrial automation device in each hand. After detecting the snapgesture performed by the user, the interactive AR system may modify avisualization associated with the AR environment to couple (e.g., snap)the two virtual industrial automation devices together at one or morepredetermined connection points. For example, the interactive AR systemmay modify the visualization associated with the AR environment toprovide a snapping motion between the two virtual industrial automationdevices as the two virtual industrial automation devices are coupledtogether. In one embodiment, the interactive AR system may provide asnapping sound that may accompany the snapping motion displayed via thevisualization associated with the AR environment. In another embodiment,the interactive AR system may determine a compatibility between the twovirtual industrial automation devices before modifying the visualizationassociated with the AR environment to snap the devices together. Forexample, the interactive AR system may display an error message afterdetermining that the two virtual industrial automation devices are notcompatible with each other (e.g., the two virtual industrial automationdevices may not couple together in the real-world).

After the interactive AR system has mapped a joint virtual industrialautomation device (e.g., two or more virtual industrial automationdevices that have been coupled together) to the hands of a user, theinteractive AR system may detect a separate gesture performed by theuser that involves the user pulling both of the user's hands apart. Forexample, the separate gesture may involve the user separating the user'shands while the user is grasping a different section of the jointvirtual industrial automation device. After detecting the separategesture performed by the user, the interactive AR system may decouple orseparate the virtual industrial automation devices grasped by the user.That is, the interactive AR system may modify a visualization associatedwith the AR environment by displaying a motion that separates thevirtual industrial automation devices apart from each other. In someembodiments, the interactive AR system may determine whether the user'shands are positioned about a line or a point of severance betweenrespective virtual industrial automation devices that are coupledtogether in the joint virtual industrial automation device. For example,the interactive AR system may determine a position of each hand of theuser along a joint virtual industrial automation device. The interactiveAR system may then determine that a line or a point of severanceassociated with the joint virtual industrial automation device islocated between the positions of each user's hands along the jointvirtual industrial automation device. In some embodiments, afterdetecting that the user has performed a gaze gesture command at a jointvirtual industrial automation device, the interactive AR system maydetermine one or more severance joints between the virtual industrialautomation devices in the joint virtual industrial automation device andmodify the visualization to display the one or more determined severancejoints. After determining that the line or the point of severance is notbetween the positions of each user's hands along the joint virtualindustrial automation device, the interactive AR system may display anerror message associated with the determination.

The interactive AR system may also detect a push gesture or a pullgesture performed by the user that involves the user placing one or bothhands on a virtual surface of a virtual industrial automation device andthe user moving in a certain direction within the visualizationassociated with the AR environment. For example, the push gesture mayinvolve the user placing both hands on a virtual surface of a virtualindustrial automation device and walking in a forward direction relativeto the position of the user. Similarly, the pull gesture may involve theuser placing both hands on a virtual surface of a virtual industrialautomation device and walking in a backward direction relative to theposition of the user. After the interactive AR system detects either thepush gesture or the pull gesture performed by the user, the interactiveAR system may move the virtual industrial automation device in adirection and at a speed that corresponds to the direction and the speedat which the user is moving. That is, the interactive AR system maymodify a visualization associated with the AR environment by displayinga continuous movement of the virtual industrial automation device in thedirection and the speed at which the user is walking in the ARenvironment. In this way, the interactive AR system may simulate apushing action against the virtual surface of the virtual industrialautomation device to move the virtual industrial automation device toanother position and a pulling action from the virtual surface of thevirtual industrial automation device to move the virtual industrialautomation device to another position in the visualization associatedwith the AR environment.

In another embodiment, the interactive AR system may detect a nudgegesture (e.g., movement of hands across some space within a certainamount of time) performed by the user that involves the user placing onehand on a virtual surface of a virtual industrial automation device andthe user moving the user's arm or hand in a certain direction within thevisualization associated with the AR environment. For example, the nudgegesture may involve the user placing a hand on a virtual surface of thevirtual industrial automation device and a movement of the user's handand/or arm in the forward direction while the user remains standing inplace. After the interactive AR system detects the nudge gestureperformed by the user, the interactive AR system may move the virtualindustrial automation device in the forward direction and at the speedthat the user's hand and/or arm are moving. That is, the interactive ARsystem may modify a visualization associated with the AR environment bydisplaying a movement of the virtual industrial automation device in theforward direction and at the speed that the user's hand and/or arm aremoving. Although the description of the nudge gesture provided above ismade in reference to moving the virtual industrial automation device inthe forward direction, it should be noted that in other embodiments, thenudge gesture may move the virtual industrial automation device in aleft direction, a right direction, a backward direction, a downwarddirection, an upward direction, or any other suitable direction based onthe direction the user's arm and/or hand is moving.

The interactive AR system may also detect a rotate gesture performed bythe user that involves the user placing a hand about an axis of rotationof the virtual industrial automation device and twisting the wrist ofthe user. After the interactive AR system detects the rotate gestureperformed by the user, the interactive AR system may rotate the virtualindustrial automation device about the axis of rotation of the virtualindustrial automation device at a speed and an angle corresponding tothe speed and the angle of rotation of the user's wrist. That is, theinteractive AR system may modify a visualization associated with an ARenvironment to display a rotating motion of the virtual industrialautomation device in the hand of the user.

As such, the interactive AR system may detect various gestures that auser may perform to assist in the design of an industrial system in anAR environment. The gestures may correspond to respective interactionswith one or more virtual industrial automation devices that may bedisplayed in a visualization associated with the AR environment. In thisway, the user may model various configurations of virtual industrialautomation devices that a user may include in the design of theindustrial system without having to physically move and interact withthe actual counterpart devices in the real world. Additionally, byperforming one or more of the various gestures described herein, a userof the interactive AR system may interact with the virtual industrialautomation devices in a visualization associated with the AR environmentin a natural and intuitive manner. That is, the interactive AR systemmay facilitate a user's interaction with the virtual industrialautomation devices in the AR environment similar to how the user wouldinteract with their counterparts in the real world. Additionally, themovement of the virtual industrial automation devices in the ARenvironment, the interactions of the virtual industrial automationdevices with other virtual objects in the AR environment, theinteractions of the virtual industrial automation devices with thephysical surroundings of the user, or the like, may obey the physicallaws of nature and simulate real-world, physical behaviors and actions.Further, the user's interactions with the virtual industrial automationdevices may ignore one or more physical laws. For example, the user maymove a virtual industrial automation device in the AR environment asthough the virtual industrial automation device is weightless andfrictionless. In this way, the interactive AR system may facilitate auser's interaction with and movement of a virtual industrial automationdevice at will and without any encumbrances but may provide a physicalsimulation of how the physical counterpart device to the virtual devicewould behave in the real-world.

Additionally, in some embodiments, the interactive AR system may detectvoice commands issued by the user to provide similar interactions oradditional interactions with a virtual industrial automation device inthe AR environment or with the AR environment itself. In someembodiments, for one or more of the gesture-based commands describedherein, a corresponding voice command may be issued by the user toperform a similar interaction with one or more virtual industrialautomation devices. For example, the user may perform a gaze gesture ata virtual industrial automation device in the AR environment and may saythe voice command “grasp.” After the interactive AR system detects thegaze gesture and the voice command, the interactive AR system may mapthe virtual industrial automation device to a hand of the user. Inanother example, after the interactive AR system maps a joint virtualindustrial automation device to a user's hands, the user may say thevoice command “separate” to cause the joint virtual industrialautomation device to decouple at a point or line of severance betweenrespective virtual industrial automation devices.

In some embodiments, a user may wish to design an industrial system froma remote location away from the physical location that the industrialsystem may be located after assembly. For example, a user may design anindustrial system from an office or in another country. In suchembodiments, the interactive AR system may provide an operational mode(e.g., dynamic rotation mode) that facilitates the design of anindustrial system in a virtual environment. That is, the interactive ARsystem may display a visualization associated with a virtual environmentthat corresponds to a scaled version of the physical location that theindustrial system may be located after assembly. The interactive ARsystem may then facilitate the user's navigation of the visualizationassociated with the virtual environment by detecting one or more gestureand/or voice commands that correspond to respective navigational toolsthat the user may employ in the visualization associated with thevirtual environment.

Additionally, the interactive AR system may facilitate a user'sinteraction with various virtual industrial automation devices in thevisualization associated with the virtual environment by detecting oneor more gestures or voice commands as described herein. For example, theinteractive AR system may detect a grasp gesture by a user and map avirtual industrial automation device to the user's hand. In anotherexample, the interactive AR system may issue a voice command to theinteractive AR system to “turn right.” After the interactive AR systemdetects the voice command, the interactive AR system may modify avisualization associated with the virtual environment to display a viewsimilar to a view that the user would perceive in the real-worldlocation after the user had turned right from the user's startingposition. As such, the interactive AR system may provide the user with avariety of design tools that facilitate a user to flexibly andconveniently design an industrial system.

In some embodiments, the interactive AR system may provide the user withan operational mode (e.g., hover mode) that provides informationassociated with one or more industrial automation devices in an existingindustrial system. That is, the user may be physically located within anindustrial system that has already been designed and assembled. The usermay perform a gaze gesture or a voice command at an industrialautomation device in the industrial system. After the interactive ARsystem detects the gaze gesture and/or the voice command by the user,the interactive AR system may display identification information,maintenance information, operational information, performanceinformation, or the like, associated with the industrial automationdevice in a visualization associated with an AR environment. Forexample, the information associated with the industrial automationdevice may be superimposed upon or adjacent to the real-worldrepresentation of the industrial automation device in the visualizationassociated with the AR environment.

With this in mind, the presently disclosed embodiments include aninteractive AR system that may be used to design an industrial system inan AR environment or provide information associated with industrialautomation devices in an existing industrial system. In someembodiments, the interactive AR system may be equipped with one or moreimage devices that may detect various gestures performed by a user tointeract with virtual representations of parts of an industrial systemdisplayed within an AR environment. Additionally, the interactive ARsystem may be equipped with one or more audio devices that may detectvarious commands issued by a user to interact with virtualrepresentations of parts of an industrial system within an ARenvironment or with the AR environment itself. Additional detailsregarding the interactive AR system and various systems and methods fordisplaying or modifying a visualization associated with the ARenvironment are described in more detail with reference to FIGS. 1-21.

By way of introduction, FIG. 1 is a block diagram of an interactive ARsystem 100 that may be utilized by a user 104 to display a visualization114 that includes a virtual representation of an industrial automationdevice 102 (e.g., virtual industrial automation device) in an ARenvironment. In the illustrated embodiment, the augmented reality (AR)environment may refer to a visualization 114 of a combination ofcomputer-generated and real-world content displayed to a user 104 via ahead mounted device 106 of the interactive AR system 100. Although ahead mounted device 106 is employed within the illustrated embodiment ofthe interactive AR system 100, it should be noted that, in otherembodiments, other suitable types of displays may be employed by theinteractive AR system 100. For example, the interactive AR system 100may employ smart glasses, a virtual retinal display, one or more contactlenses, a computer, a mobile device, or any other suitable electronicdisplay device for displaying visualizations to a user. In any case, thehead mounted device 106 may display a visualization 114 that includes avirtual industrial automation device 102 to the user 104. Thevisualization 114 may be superimposed computer-generated content (e.g.,images or sounds) over real-world content (e.g., images or sounds) ofthe user's environment in real-time. Additional details with regard tothe head mounted device 106 may be discussed below with reference toFIG. 2.

In the illustrated embodiment, the interactive AR system 100 may displaya visualization 114 via the head mounted device 106 that includes avirtual representation of a motor drive 102. However, it should be notedthat the illustrated embodiment is intended to be non-limiting and thatthe interactive AR system 100 may display a visualization 114 via thehead mounted device 106 that may include other virtual industrialautomation devices, or parts thereof, that may be employed within anindustrial system. For example, the industrial automation devices mayinclude controllers, input/output (I/O) modules, motor control centers,motors, valves, actuators, temperature elements, pressure sensors, humanmachine interfaces (HMIs), operator interfaces, contactors, starters,sensors, drives, relays, protection devices, switchgear, compressors,network switches (e.g., Ethernet switches, modular-managed,fixed-managed, service-router, industrial, unmanaged, etc.), datacenters, conveyor sections, movers, and the like.

In certain embodiments, the head mounted device 106 of the interactiveAR system 100 may detect a gesture command performed by a user 104. Forexample, the interactive AR system 100 may detect a gaze gestureperformed by the user 104 directed at a virtual industrial automationdevice 102 to request information or data associated with the industrialautomation device 102. The head mounted device 106 may analyzecharacteristics of image data associated with the user's biomechanicalmovements to determine if the image data matches a characteristic of agesture command stored, learned, or otherwise interpretable by the headmounted device 106 of the interactive AR system 100. Image dataassociated with the user's biomechanical movements may include themotion, or lack thereof, of the user's hands, wrists, arms, fingers, orany other suitable body part to distinguish one gesture command fromanother gesture command. In some embodiments, the head mounted device106 may acquire the image data and send the image data, via network 108,to a computing system 110 to analyze the characteristics of the imagedata to determine if the image data matches a characteristic of agesture command stored, learned, or otherwise interpretable by thecomputing system 110.

In some embodiments, the head mounted device 106 may be communicativelycoupled to one or more motion sensors attached to a user's body. Forexample, one or more motion sensors may be disposed on the user's hands,wrists, arms, fingers, legs, feet, torso, or any other suitable bodypart and provide motion data (e.g., body motion capture data) to thehead mounted device 106. In one embodiment, based on the received motiondata associated with the user 104, the head mounted device 106 mayanalyze the motion data associated with a respective body part of theuser 104 and determine a gesture command stored, learned, or otherwiseinterpretable by the head mounted device 106. In another embodiment, thehead mounted device 106 may analyze the motion data associated with therespective body part of the user 104 and determine a virtual force(e.g., a virtual speed, virtual displacement, or virtual direction)associated with a gesture command performed by the user. For example,the head mounted device 106 may determine a speed and an angleassociated with the movement of the user's hand or foot after the user104 performs a push gesture command against a virtual industrialautomation device. The head mounted device 106 may then modify avisualization 114 to display an animation of a movement of the virtualindustrial automation device based on the determined speed and angleassociated with the movement of the user's hand or foot.

In the illustrated embodiment, the computing system 110 may becommunicatively coupled to a database 112 that may store a list ofgesture commands that are learned or otherwise interpretable by the headmounted device 106 and/or the computing system 110. The database 112 mayalso store a list of user profiles that include gesture commands thatmay correspond to specific users 104 that are learned or otherwiseinterpretable by the head mounted device 106 and/or the computing system110. For example, the head mounted device 106 and/or the computingsystem 110 may retrieve a user profile that includes a list of gesturecommands that corresponds to the specific user 104 utilizing the headmounted device 106. The head mounted device 106 and/or the computingsystem 110 may analyze characteristics of the image data to determine ifthe image data matches a characteristic of the received gesture commandsof the user 104. In some embodiments, if a threshold of one or morecharacteristics for a gesture command or a verbal command match astored, learned, or otherwise interpretable gesture command, the headmounted device 106 and/or the computing system 110 may determine that agesture command has been performed by the user 104 of the head mounteddevice 106 based on the image data.

It should be noted that any suitable network may be employed in theembodiments described herein. For instance, the network 108 may includeany wired or wireless network that may be implemented as a local areanetwork (LAN), a wide area network (WAN), and the like. Indeed, otherindustrial communication network protocol, such as EtherNet/IP,ControlNet, DeviceNet, and the like, may also be used. In any case, thenetwork 108 may permit the exchange of data in accordance with aprotocol.

After detecting a gesture performed by the user 104, the head mounteddevice 106 of the interactive AR system may request information or dataassociated with a virtual industrial automation device 102 from thecomputing system 110 communicatively coupled to the head mounted device106 via the network 108. The computing system 110 may then send arequest to a database 112 communicatively coupled to the computingsystem 110 for the information or the data associated with theindustrial automation device 102. In some embodiments, the computingsystem 110 and the database 112 may be part of the same device.Additionally, it should be noted that the computing system 110 may beany suitable computing device that includes communication abilities,processing abilities, and the like. For example, the computing system110 may be any general computing device that may communicate informationor data to the head mounted device 106 via the network 108.

The type of information or data associated with the industrialautomation device 102 and requested by the head mounted device 106 maybe based on the gesture performed by the user 104 and detected by thehead amounted device 106. In one embodiment, the head mounted device 106may detect a gesture command (e.g., a gaze gesture) performed by a user104 to select a virtual industrial automation device 102 to furtherinteract with (e.g., move, rotate, scale up, or scale down) in avisualization 114 associated with the AR environment. The head mounteddevice 106 may send a request to the computing system 110 forspecification data associated with the virtual industrial automationdevice 102. For example, the specification data may include a virtualphysics dataset associated with the industrial automation device 102.For example, the virtual physics dataset may include a virtual weight ofthe industrial automation device 102, virtual dimensions of theindustrial automation device 102, and the like.

After receiving the virtual physics dataset associated with theindustrial automation device 102, the head mounted device 106 maysimulate the real-world physical characteristics of the industrialautomation device 102 in the visualization 114 associated with the ARenvironment via the virtual industrial automation device 102. Thespecification data may also include other visual data associated withthe industrial automation device 102, such as possible color schemes, orthe like. As such, the head mounted device may receive specificationdata associated with the virtual industrial automation device 102 fromthe computing system 110. Based on the received specification data, thehead mounted device 106 may generate and display a virtual industrialautomation device 102 in the visualization 114 associated with the ARenvironment to the user 104.

In another embodiment, the user 104 may utilize the head mounted device106 to obtain operational data or maintenance data regarding theindustrial system. After the head mounted device 106 of the interactiveAR system 100 detects a gesture command (e.g., a gaze gesture command)performed by the user 104 to select at an actual industrial automationdevice in the industrial system, the head mounted device 106 may requestvarious types of data (e.g., identification data, operational data, ormaintenance data) associated with the industrial automation device 102from the computing system 110 to display to a user in a visualization114 associated with the AR environment. For example, the identificationdata may include a product name, a product type, a vendor name, a cost,a description of the function of the industrial automation device, orthe like. The operational data may include data gathered by one or moresensors in the industrial system that measure one or more operationalparameters associated with the industrial automation device 102. Themaintenance data may include data associated with maintenance recordsand/or data logs of the industrial automation device 102. As such, thehead mounted device 106 may receive various types of data associatedwith a real-world industrial automation device in an industrial systemfrom the computing system 110 and display the data to the user 104 in avisualization 114 associated with the AR environment.

In another embodiment, the user 104 may perform a gesture command (e.g.,a gaze gesture command) to select a virtual industrial automation device102 when designing an industrial system to obtain identificationinformation associated with the industrial automation device 102. Forexample, after the head mounted device 106 of the interactive AR system100 detects a gaze gesture performed by the user 104 and directed at thevirtual representation of the industrial automation device 102, the headmounted device 106 may request identification data associated with theindustrial automation device 102 from the computing system 110 todisplay to the user 104 in the visualization 114 associated with the ARenvironment. For example, the identification data may include a productname, a product type, a vendor name, a cost, a description of thefunction of the industrial automation device, or the like. As such, thehead mounted device 106 may receive identification data associated witha virtual industrial automation device 102 from the computing system 110and display the identification data to user in the visualization 114associated with the AR environment.

In another embodiment, the user 104 may perform a snap gesture commandto couple a first virtual industrial automation device and a secondvirtual industrial automation device together. After the head mounteddevice 106 of the interactive AR system 100 detects the snap gesture bythe user 104, the head mounted device 106 may request compatibility dataassociated with the first virtual industrial automation device and thesecond virtual industrial automation device from the computing system110. For example, the compatibility data may include a first list ofdevices that are compatible with the first industrial automation deviceand a second list of devices that are compatible with the secondindustrial automation device. As such, the head mounted device 106 mayreceive compatibility data associated with the industrial automationdevice 102 from the computing system 110. Based on the receivedcompatibility data, the head mounted device 106 may then determinewhether the first industrial automation device and the second industrialautomation device are compatible and display a notification associatedwith the determination.

As described above, the head mounted device 106 may request informationor data associated with an industrial automation device 102 from thecomputing system 110 that is communicatively coupled to the database112. The database 112 may be organized to include a list of variousindustrial automation devices that may be employed in the design of anindustrial system. In some embodiments, the database 112 may index thedata associated with the industrial automation device 102 based on anidentifier associated with each industrial automation device 102. Insuch embodiments, the head mounted device 106 may identify an identifierof an industrial automation device 102 based on the gesture command orvoice command performed by the user 104 and data associated with theindustrial automation device 102. The head mounted device 106 may thensend a request with the identifier associated with the industrialautomation device 102 to the computing system 110. The computing system110 may then extract data associated with the industrial automationdevice 102 based on the identifier and/or the type of request and sendthe extracted data to the head mounted device 106. After the headmounted device 106 receives the data associated with the industrialautomation device 102, the head mounted device 106 may generate anddisplay a visualization 114 that includes a visual representation of thedata associated with the industrial automation device 102.

To perform some of the actions set forth above, the head mounted device106 may include certain components to facilitate these actions. FIG. 2is a block diagram 200 of exemplary components within the head mounteddevice 106. For example, the head mounted device 106 may include one ormore cameras 202 and one or more microphones 204. It should beunderstood tat any suitable image-receiving device may be used in placeof, or in addition to, the cameras 202, for example, a singular camera202 may be incorporated into the head mounted device 106. It also shouldbe understood that any suitable sound-receiving device may be used inplace of, or in addition to, the microphones 204, for example, acombined speaker and microphone device, or a singular microphone 204 maybe incorporated into the head mounted device 106.

In some embodiments, the head mounted device 106 may include one or moresensors for detecting the movements of the user 104, the biometrics ofthe user 104, the surroundings of the user 104, or the like. Forexample, the head mounted device 106 may include an infrared sensor, athermal sensor, a range sensor (e.g., a range camera), a smell sensor(e.g., an electronic nose), or any other suitable sensors for detectingcharacteristics of the user 104 or the surroundings of the user 104.

The head mounted device 106 may also include processing circuitry 206including a processor 208, a memory 210, a communication component 212,input/output (I/O) ports 214, and the like. The communication component212 may be a wireless or a wired communication component that mayfacilitate communication between the head mounted device 106 and thecomputing system 110, the database 112, and the like via the network108. This wired or wireless communication protocols may include anysuitable communication protocol include Wi-Fi, mobile telecommunicationstechnology (e.g., 2G, 3G, 4G, LTE), Bluetooth®, near-fieldcommunications technology, and the like. The communication component 212may include a network interface to enable communication via variousprotocols such as EtherNet/IP®, ControlNet®, DeviceNet®, or any otherindustrial communication network protocol.

The processor 208 of the head mounted device 106 may be any suitabletype of computer processor or microprocessor capable of executingcomputer-executable code, including but not limited to one or more fieldprogrammable gate arrays (FPGA), application-specific integratedcircuits (ASIC), programmable logic devices (PLD), programmable logicarrays (PLA), and the like. The processor 208 may, in some embodiments,include multiple processors. The memory 210 may include any suitablearticles of manufacture that serve as media to storeprocessor-executable code, data, and the like. The memory 210 may storenon-transitory processor-executable code used by the processor 208 toperform the presently disclosed techniques.

Generally, the head mounted device 106 may receive image data or audiodata related to a user 104 via one or more image sensors (e.g., cameras202) or one or more audio sensors (e.g., microphones 204), respectively,communicatively coupled to one or more of the I/O ports 214. Uponreceiving image data or audio data, the head mounted device 106, via theprocessor 208, may interpret the image data or the audio data todetermine commands or actions for the head mounted device 106 to performin response to the determined commands or actions. In some embodiments,the determined command may be forwarded to computing system 110 tointerpret the detected image data or audio data. The computing system110 may analyze characteristics of the image data or the audio data todetermine if the image data or the audio data matches the characteristicof a gesture command or verbal command, respectively, stored, learned orotherwise interpretable by the computing system 110.

As mentioned above, the database 112 may store a list of gesturecommands or voice commands that are stored, learned, or otherwiseinterpretably by the computing system 110. For example, the list ofgesture or voice commands may include a snap gesture command, a separategesture command, a push gesture command, a pull gesture command, arotate gesture command, a nudge gesture command, a lift gesture command,a let-go gesture command, a grasp gesture command, a gaze gesturecommand, a scale up gesture command, a scale down gesture command, orthe like. In another embodiment, instead of forwarding the command tothe computing system 110, the head mounted device 106 may be able toanalyze characteristics of the image data or the audio data to determineif the image data or the audio data matches the characteristic of agesture command or verbal command, respectively, stored, learned, orotherwise interpretable by the head mounted device 106. In any case, thehead mounted device 106 or the computing system 110 may analyzecharacteristics of the user's movements in the image data such as themotion of the user's hands, wrists, arms, fingers, or any other suitablebody part to distinguish one gesture command from another gesturecommand.

Additionally, the head mounted device 106 or the computing system 110may analyze characteristics of the audio data, such as frequency (e.g.,pitch), amplitude (e.g., loudness), or any other suitable characteristicused to distinguish one verbal command from another verbal command. If athreshold of one or more characteristics for a gesture command or averbal command match a stored, learned, or otherwise interpretablecommand, the head mounted device 106 may determine a command to beperformed by the head mounted device 106 based on the image data or theaudio data.

As discussed above, the head mounted device 106 may be communicativelycoupled to the network 108, which may include an Internet connection, orotherwise suitable wireless or wired communicative coupling to expandits interpretation and functional capabilities, but, in someembodiments, the head mounted device 106 may not rely on such acommunicative coupling. In other words, the head mounted device 106 mayhave particular capabilities that may function without an Internet,wireless, or wired connection. For example, the head mounted device 106may perform local command interpretation without an Internet or wirelessconnection.

The head mounted device 106 may also include a video output 216. Thevideo output 216 may be any suitable image-transmitting component, suchas a display. Head mounted device 106 may display a visualization 114associated with the AR environment that combines computer-generatedcontent, such as a virtual industrial automation device 102, withreal-world content, such as image data associated with the user'sphysical surroundings.

With the foregoing in mind, FIG. 3 illustrates a perspective view 300 ofa user 104 utilizing a head mounted device 106 to perceive thevisualization 114 associated with an AR environment. By way of example,in the illustrated embodiment, the user 104 may design a conveyor systemfor an industrial system. The head mounted device 106 of the interactiveAR system 100 may generate and display to the user 104 a visualization114 with virtual representations of various compartments 302, 304 (e.g.,virtual compartments) that each correspond to a type or a category ofindustrial automation device that may be employed by a conveyor system.For example, the head mounted device 106 may acquire image data of theuser and the user's physical surroundings (e.g., the real world) andgenerate and display a visualization 114 that superimposes the virtualcompartments 302, 304 over portions of the acquired image data of thereal world in real-time or substantially real-time. In some embodiments,the virtual compartments 302 may resemble real-world objects, such asboxes, storage bins, lockers, or the like, or may have a design that isnot directly tied to a real-world object.

As described above, the virtual compartments 302, 304 may correspond torespective types or categories of industrial automation devices that maybe employed within a conveyor system. For example, compartment 302 maycorrespond to different types of conveyor sections that may be employedwithin a conveyor system. In another example, compartment 304 maycorrespond to different types of movers that may be employed within aconveyor system. The user 104 may interact with the compartments 302,304 by performing a gaze gesture command directed at one of thecompartments 302, 304 to open.

In some embodiments, a gaze gesture command may be detected by the headmounted device 106 by tracking the movement of the user's eyesassociated with a virtual surface of a virtual object (e.g., a virtualcompartment or a virtual industrial automation device). For example, thehead mounted device 106 may continuously or intermittently acquire imagedata of the user's eyes and track a location on the display that theuser's eyes are focusing on (e.g., a virtual object). In otherembodiments, a visual indicator (e.g., dot) displayed in thevisualization 114 may correspond to a cursor that the user 104 may useto focus a gesture command on within the visualization 114 associatedwith the AR environment. For example, the user may change the positionof the dot by moving the user's head left, right, up, or down. Once thecursor is positioned on a certain object for greater than a thresholdperiod of time (e.g., greater than 3 seconds, 4 seconds, 5 seconds, orthe like), the head mounted device 106 may detect that the user hasperformed a gaze gesture command to select the object within thevisualization 114 associated with the AR environment.

In any case, after the head mounted device 106 has detected that theuser 104 has performed a gaze gesture command directed at a virtualcompartment 302, 304, the head mounted device 106 may modify thevisualization 114 to display one or more virtual industrial automationdevices corresponding to the virtual compartment selected by the user104 via the gaze gesture command. To help illustrate, FIG. 4 illustratesa perspective view 400 of the user 104 utilizing the head mounted device106 to perceive a modified visualization 114 associated with the ARenvironment illustrated in FIG. 3. In the illustrated embodiment, thehead mounted device 106 of the interactive AR system 100 may display oneor more types of virtual conveyor sections 402, 404, 406. For example,virtual conveyor section 402 may have a curved shape, virtual conveyorsection 404 may have a straight shape, and virtual conveyor section 406may have a straight shape and a curved shape.

Although FIGS. 3 and 4 illustrate that the head mounted device 106 maydisplay two virtual categorical compartments 302, 304 of virtualindustrial automation devices with a category having three types 402,404, 406 of a particular industrial automation device, it should benoted that in other embodiments, the virtual categorical compartmentsand the virtual industrial automation devices in each category are notlimited as such in number. In some embodiments, the head mounted device106 may display any number of virtual compartments 302, 304 and/orvirtual industrial automation devices 402, 404, 406 in the visualization114 associated with the AR environment. For example, the head mounteddevice 106 may display one, two, five, ten, twenty, fifty, one hundred,or any other suitable number of virtual compartments and/or virtualindustrial automation devices in the visualization 114 associated withthe AR environment. In some embodiments, the user 104 may look or tiltthe user's head to the left, right, up, down, or the like, and the headmounted device 106 may display additional virtual compartments and/orvirtual industrial automation devices in the visualization 114associated with the AR environment accordingly. In one embodiment, theuser 104 may resize the virtual compartments and/or virtual industrialautomation devices in the visualization 114 such that the availablevirtual compartments and/or virtual industrial automation devices fordisplay to the user 104 may accommodate the user's visual needs. Forexample, the user 104 may perform a gesture command (e.g., scale down)or a voice command (e.g., say “scale down”) to decrease the size of thevirtual compartments and/or virtual industrial automation devices in thevisualization 114 such that the user 104 may be able to view and selectfrom tens of options, hundreds of options, thousands of options, or thelike, without the user 104 having to look in a different direction orturn the user's head in a different direction to view additional virtualcompartments and/or virtual industrial automation devices in thevisualization 114.

In another embodiment, the head mounted device 106 may modify thevisualization 114 associated with the AR environment and displayadditional and/or different virtual compartments and/or virtualindustrial automation devices as the user 104 moves through the user'ssurroundings. For example, the head mounted device 106 may display afirst subset of virtual industrial automation devices in thevisualization 114 associated with the AR environment to the user 104while the user is in a first position in the user's surroundings. Afterthe head mounted device 106 determines that the user 104 has moved to asecond position (e.g., greater than a certain threshold associated withdisplaying one or more additional virtual industrial automationdevices), the head mounted device 106 may display a second subset ofvirtual industrial automation devices in the visualization 114associated with the AR environment. In another embodiment, the firstsubset of the virtual industrial automation devices and the secondsubset of virtual industrial automation devices may be displayed to theuser 104 while the user is in the first position but the second subsetof virtual industrial automation devices may be displayed in aproportional smaller size with respect to the first subset of virtualindustrial automation devices to simulate that the second subset isfurther away from the user 104 in the AR environment. As the user 104moves toward the second subset of the virtual industrial automationdevices in the AR environment, the head mounted device 106 may modifythe visualization 114 to display the second subset of the virtualindustrial automation devices increasing in size as the user walkstoward the second subset and the first subset of the virtual industrialautomation devices decreasing in size as the user 104 walks away fromthe first subset. That is, the head mounted device 106 may modify thevisualization 114 to simulate the perspective of the user 104 walkingaway from or toward various real-world industrial automation devices inthe user's surroundings.

In other embodiments, the number of virtual compartments 302, 304 and/orthe number of virtual industrial automation devices 402, 404, 406 thatmay be displayed to a user 104 in the visualization 114 may be limitedby the area of the display of the visualization 114. For example, thenumber of virtual compartments 302, 304 or the number of virtualindustrial automation devices 402, 404, 406 that may be displayed by thehead mounted device 106 in the visualization 114 of the AR environmentmay be more than two, more than five, more than ten, more than twenty,or the like, based on the size of the virtual compartments 302, 304 orthe virtual industrial automation devices 402, 404, 406 in thevisualization 114 and the display area of the visualization 114.

In some embodiments, the head mounted device 106 may display a subset ofthe number of virtual compartments 302, 304 or a subset of the number ofvirtual industrial automation devices 402, 404, 406 based on the spaceavailable via the visualization 114. For example, the head mounteddevice 106 may display a visualization 114 with four of twenty virtualcompartments 302, 304 that correspond to respective types of virtualindustrial automation devices 402, 404, 406. The user may then perform agesture command or issue a voice command to that may cause the headmounted device 106 to modify the visualization 114 to display the nextfour virtual compartments 302, 304 of the twenty virtual compartments302, 304. For example, the head mounted device 106 may detect a swipinggesture command (e.g., hand swiping across the virtual compartments 302,304) performed by the user 104 to display the next subset of virtualcompartments 302, 304. In another example, the head mounted device 106may detect a “next” voice command issued by the user to display the nextsubset of virtual compartments 302, 304. In another example, the numberof virtual compartments 302, 304 displayed in the visualization 114 maynot be limited by the display area of the visualization 114. That is,the head mounted device 106 may display additional virtual compartments302, 304 after detecting that the user has turned the user's headupward, downward, to the left, to the right, and so forth.

Referring back to FIG. 4, the head mounted device 106 may also display avirtual representation of the conveyor system 408 (e.g., a virtualconveyor system) that has been designed or partially designed by theuser 104. For example, the virtual conveyor system 408 may include oneor more conveyor shapes that have been placed by the user in a specificconfiguration in the visualization 114 associated with the ARenvironment. The user 104 may be able to determine a desired shape ofconveyor section 402, 404, 406 to employ in the design of the conveyorsystem 408 based on the display of the virtual conveyor system 408. Forexample, the user 104 may be able to determine that the specificconfiguration of the virtual conveyor system 408 is missing a curvedconveyor section. The user may then perform a gaze gesture commanddirected at one of the virtual industrial automation devices 404corresponding to the curved conveyor section. After detecting the gazegesture command performed by the user 104, the head mounted device 106may generate and display a modified visualization 114 that displays theuser's selection of the virtual industrial automation device 404.

In some embodiments, based on the configuration of the virtual conveyorsystem 408, the head mounted device 106 may modify the visualization 114to display a subset of available virtual industrial automation devicesthat may be utilized with the configuration of the virtual conveyorsystem 408. For example, as the user 104 places virtual industrialautomation devices to form the virtual conveyor system 408, the headmounted device 106 may display a smaller subset of virtual industrialautomation devices that correspond to virtual industrial automationdevices that may couple to the user-placed virtual industrial automationdevices in the virtual conveyor system 408. In this way, the headmounted device 106 may predict one or more virtual industrial automationdevices that the user 104 may desire to select and place next based onone or more previous selections and placements of virtual industrialautomation devices performed by the user 104.

Further, in one embodiment, the head mounted device 106 may determinethat the user may desire another type of industrial automation deviceafter the selection and placement of a first type of virtual industrialautomation device. The head mounted device 106 may display one or morevirtual compartments that correspond to the other types of industrialautomation devices that the user 104 may desire after selecting andplacing the first type of virtual industrial automation device. Forexample, the head mounted device 106 may determine that the user 104 hasfinished designing the track of a virtual conveyor system. The headmounted device 106 may modify the visualization 114 to display a virtualcompartment associated with various types of virtual movers that may becoupled or placed on the track of the virtual conveyor system. In someembodiments, the head mounted device 106 may modify the visualization114 to display both virtual industrial automation devices and virtualcompartments associated with predicted selections by the user 104 afterthe user 104 has selected and placed a virtual industrial automationdevice.

In some embodiments, as shown in FIG. 4, virtual industrial automationdevices 402, 404, 406 that have not yet been selected by the user 104 inthe visualization 114 associate with the AR environment may beillustrated in dotted lines. After the head mounted device 106 detects agesture command performed by the user 104 to select a virtual industrialautomation device 402, 404, 406, the head mounted device 106 may modifythe visualization 114 to display the appearance of the selected virtualindustrial automation device in solid lines. It should be noted that thedescription of the transition between dotted lines and solid lines fordisplaying not selected and selected virtual industrial automationdevices, respectively, is exemplary and non-limiting. Additionally,other embodiments may include transitions in appearance from notselected to selected virtual industrial automation devices, such as ahighlighting, a color change, a shading, or any other suitable visualchange in appearance.

FIG. 5 illustrates a perspective view 500 of the user 104 utilizing thehead mounted device 106 to perceive a modified visualization 114associated with the AR environment illustrated in FIGS. 3 and 4. Afterdetecting a gaze gesture command performed by the user 104, the headmounted device 106 may determine that the user 104 selected the firstvirtual conveyor section 402, as shown in FIG. 4. The head mounteddevice 106 may then generate and display a modified visualization 114corresponding to the user's selection of the first virtual conveyorsection 402. For example, the head mounted device 106 may display thefirst virtual conveyor section 402 in solid lines in the modifiedvisualization 114 to indicate that the first virtual conveyor section402 has been selected by the user 104 and/or may now be interacted withby the user in the AR environment.

With the foregoing in mind, FIG. 6 illustrates a flow chart of a method600 for displaying and modifying a visualization 114 associated with anAR environment based on one or more gaze gestures commands performed bya user 104. Although the following description of the method 600 isdescribed in a particular order, it should be noted that the method 600is not limited to the depicted order, and instead, the method 600 may beperformed in any suitable order. Moreover, although the method 600 isdescribed as being performed by the head mounted device 106, it shouldbe noted that it may be performed by any suitable computing devicecommunicatively coupled to the head mounting device 106.

Referring now to FIG. 6, at block 602, the head mounted device 106 mayreceive image data of the physical space associated with the real-worldenvironment of the user 104. In some embodiments, the head mounteddevice 106 may acquire image data via the one or more cameras 202. Theimage data may include data that indicates dimensions of the physicalspace, such as height, width, and length. The head mounted device 106may then process the acquired image data and display the visualization114 based on the image data with respect to the physical spaceassociated with the real-world environment of user 104. For example, thecameras 202 may acquire image data of real-world objects withinsurrounding environment of the user 104. The real-world objects mayinclude physical structures, the user's body, other real-world objects,or portions thereof.

At block 604, the head mounted device 106 may generate and display avisualization 114 based on the acquired image data. For example, thevisualization 114 may replicate the acquired image data on a display ofthe head mounted device 106. In certain embodiments, the head mounteddevice 106 may generate and display the visualization 114 to simulatethe user's perception of the physical space associated with real-worldenvironment of the user 104. For example, the head mounted device 106may generate and display the visualization 114 to have the same viewingangle, the same field of vision, the same depth of vision, or the like,that the user 104 may perceive of the real-world surrounding the user.Alternatively, the visualization 114 may be presented via a transparentdisplay that allows the user 104 to view the real world surroundings.The visualization 114 may then be superimposed over the transparentdisplay to produce virtual objects within the real world surroundings.In some embodiments, the head mounted device 106 may provide a videosee-through display or an optical see-through display to displayvisualizations to the user.

In some embodiments, the head mounted device 106 may display a virtualcompartment 302, 304 that corresponds to a type or category ofindustrial automation device that may be employed within an industrialsystem. For example, a virtual compartment 302 may correspond to one ormore types of conveyor sections that may be employed within a conveyorsystem. As such, the head mounted device 106 may display a visualization114 to a user 104 on a display that includes both real-world andcomputer-generated content in real-time or substantially real-time. Insome embodiments, the user 104 may speak a voice command to indicate thetype or category of the industrial automation system that is intended tobe placed in the physical space. Otherwise, the user 104 may specify tothe head mounted device 106 the type or category of the industrialautomation system by scrolling through visualizations that providecategories or types of industrial automation systems that may bedesigned.

At block 606, the head mounted device 106 may receive a selection of avirtual compartment 302, 304 based on a gaze gesture command performedby the user 104. For example, the head mounted device 106 may acquireimage data of the user 104 or a portion thereof, such as the user'sarms, hands, fingers, legs, or the like. The head mounted device 106 maythen detect a gesture command performed by the user 104 based on theacquired image data. As described above, for example, a gaze gesture maybe detected by the head mounted device 106 by tracking the movement ofthe user's eyes or tracking a cursor indicative of the user's focus in avisualization associated with the AR environment.

After the head mounted device 106 receives an indication of theselection of the virtual compartment 302, 304 based on the gaze gesturecommand performed by the user 104, the head mounted device 106 maymodify the visualization 114 based on the selection of the virtualcompartment 302, 304 at block 608. For example, the head mounted device106 may modify the visualization 114 to display one or more virtualindustrial automation devices 402, 404, 406 associated with the selectedvirtual compartment 302, 304. In some embodiments, the head mounteddevice 106 may send a request to the computing system 110 for a list ofindustrial automation devices stored in the database 112 associated withthe selected virtual compartment 302, 304. The head mounted device 106may also send a request to the computing system 110 for specificationdata associated with each industrial automation device associated withthe selected virtual compartment 302, 304. After receiving the list ofindustrial automation devices associated with the type or categoryrepresented by the selected virtual compartment 302, 304 and thespecification data associated with each industrial automation device,the head mounted device 106 may modify the visualization 114 to displaya virtual industrial automation device based on the specification datain the AR environment. That is, the specification data may include imageor dimensional data that may be used to generate a virtual object thatrepresents the virtual industrial automation device.

At block 610, the head mounted device 106 may receive another indicationof a selection of a virtual industrial automation device 402, 404, 406based on a gaze gesture command performed by the user 104. Similar tothe selection of the virtual compartment 302, 304 described above, thehead mounted device 106 may acquire image data of the user 104 or aportion thereof, such as the user's arms, hands, fingers, legs, or thelike. The head mounted device 106 may then detect a gaze gesture commandbased on the acquired image data.

After the head mounted device 106 receives the selection of the virtualindustrial automation device 402, 404, 406 based on the gaze gesturecommand performed by the user 104, the head mounted device 106 maymodify the visualization 114 based on the selection of the virtualindustrial automation device 402, 404, 406 at block 612. For example,the head mounted device 106 may modify the visualization 114 bydisplaying the selected virtual industrial automation device 402, 404,406. In some embodiments, the selected virtual industrial automationdevice may be displayed in solid lines as compared to dotted lines whenthe virtual industrial automation device had not yet been selected. Assuch, the head mounted device 106 may detect a gaze gesture commandperformed by a user 104 to select and display a virtual industrialautomation device 402, 404, 406 in the visualization 114 of an ARenvironment.

Additionally, in some embodiments, the head mounted device 106 maydetect voice commands issued by the user to provide similar interactionsor additional interactions with the virtual compartments 302, 304 and/orthe virtual industrial automation devices 402, 404, 406 in the ARenvironment or with the AR environment itself. For example, the user 104may look towards a virtual compartment 302, 304 and/or a virtualindustrial automation device 402, 404, 406 and may say the voice command“select.” After the head mounted device detects the voice command, thehead mounted device 106 may perform actions as described herein withrespect to the gaze gesture command.

In some embodiments, the user 104 may perform a gaze gesture command todisplay information or data associated with an industrial automationdevice 102. For example, in a design context, the user 104 may wish toknow the name, the type, the vendor, the cost, or the like, of anindustrial automation device. In another example, the user 104 may wishto know the identification, maintenance, and operational information, orthe like, associated with a particular industrial automation device inan existing industrial system. With the foregoing in mind, FIG. 7illustrates a perspective view 700 of a user 104 utilizing the headmounted device 106 to perceive a visualization 114 that may displayidentification information 702 associated with an industrial automationdevice 102. For example, the head mounted device 106 may acquire imagedata of the surroundings of the user 104 via one or more cameras 202.The head mounted device 106 may then process the acquired image data ofthe user and the user's surroundings and detect a gaze gesture commandperformed by the user 104 based on the image data. After detecting thegaze gesture command performed by the user 104, the head mounted device106 may determine a target of the gaze gesture based on the image dataand/or the gaze gesture command. As illustrated in FIG. 7, the target ofthe gaze gesture may be a virtual motor drive 102. The head mounteddevice 106 may then receive an identifier associated with the virtualmotor drive 102 after determining that the virtual motor drive 102 isthe target of the gaze gesture. For example, the head mounted device 106may retrieve an identifier stored in the memory 210 of the head mounteddevice 106 that corresponds to the virtual motor drive 102. The headmounted device 106 may then send a request with the identifier to thecomputing system 110 for identification information associated with theindustrial automation device 102. Based on the identifier and the typeof request sent by the head mounted device 106, the computing system 110may send identification information associated with the identifier tothe head mounted device 106. After receiving the identificationinformation associated with the identifier, the head mounted device 106may display a virtual representation of the identification informationon or adjacent to the virtual industrial automation device 102 in thevisualization 114 associated with the AR environment.

After the user 104 has selected a virtual industrial automation device102 within the visualization 114 associated with the AR environment, theuser 104 may wish to reposition the virtual industrial automation device102 within the visualization 114 to a desired position. To helpillustrate, FIG. 8 is a perspective view of a user 104 utilizing thehead mounted device 106 to reposition a virtual industrial automationdevice 806 in the visualization 114 associated with an AR environment.The head mounted device 106 may detect a gesture command 802 performedby the user 104 to select the virtual industrial automation device 806within the visualization 114 associated with the AR environment. In theillustrated embodiment, the gesture command 802 performed by the user104 may involve the user 104 reaching out in a direction toward adesired virtual industrial automation device 806 with, in oneembodiment, a flat or open palm over the selected virtual industrialautomation device 806. In other embodiments, the gesture command 802performed by the user to select the virtual industrial automation device806 may be a gaze gesture command as described above. Based on the imagedata, the head mounted device 106 may detect the gesture command 802performed by the user and the target of the gesture command 802. In theillustrated embodiment, the head mounted device 106 may determine avector extending along the user's arm or the user's palm toward avirtual industrial automation device 804, 806 based on the image data.The head mounted device 106 may then determine that the virtualindustrial automation device 806 is the target of the user's gesturecommand 802 because the position of the virtual industrial automationdevice 806 intersects with the vector extending from the user's arm orthe user's palm. In some embodiments, the head mounted device 106 mayalso track the eye movements of the user or track a cursor indicative ofthe focus of the user 104 in the visualization associated with the ARenvironment to determine the target of the user's gesture command 802.

FIG. 9 is a perspective view of a user 104 utilizing the head mounteddevice 106 to reposition the virtual industrial automation device 806 tothe hand of the user 104. The head mounted device 106 may detect agrasping gesture command 808 by the user 104 with the same hand used toselect the virtual industrial automation device 806 as shown in FIG. 8.As described above, the head mounted device 106 may receive image dataassociated with the user and the user's surroundings and detect thegrasping gesture command 808 performed by the user based on the imagedata. The head mounted device 106 may then identify one or more mappingpoints associated with the user's hand that performed the graspinggesture command 808. After the head mounted device 106 has detected thegrasping gesture command 808, the head mounted device 106 may modify thevisualization associated with the AR environment and map the selectedvirtual industrial automation device 806 to the user's hand thatperformed the grasping gesture 808 at the one or more identified mappingpoints. Thereafter, as the user 104 moves the user's hand in thevisualization 114 associated with the AR environment, the head mounteddevice 106 may continuously modify the visualization associated with theAR environment to move (e.g., as an animation) the selected virtualindustrial automation device 806 toward the one or more identifiedmapping points associated with the user's hand. That is, the user 104may move the virtual industrial automation device 806 in thevisualization associated with the AR environment in real-time orsubstantially real-time after performing a grasping gesture command 808in the visualization.

In one example, moving the user's hand from the open palm position tothe partially closed (e.g., u-shaped) position may be detected as agesture that causes the head mounted device 106 to move the selectedvirtual industrial automation device 806 to the hand of the user 104performing the gesture. The mapped points of the hand may include one ormore fingers, the palm, or other distinguishable features of the hand.When the virtual industrial automation device 806 is selected and thegrasp gesture is initialized, the head mounted device 106 may cause thevirtual industrial automation device 806 to move (e.g., as an animation)towards the mapped points and stay attached to the mapped points untilanother gesture or voice command is received.

After the user 104 has grasped and/or repositioned a virtual industrialautomation device 102 within the visualization 114 associated with theAR environment, the user 104 may wish to drop the virtual industrialautomation device 102 within the visualization 114. The head mounteddevice 106 may detect a let go gesture command (e.g., a release gesturecommand) performed by the user 104 to release the virtual industrialautomation device 806 from the user's hand within the visualization 114associated with the AR environment. For example, the let go gesturecommand may involve the user 104 extending the user's fingers from acurled position around the virtual industrial automation device 806.Based on the image data associated with the user 104, the head mounteddevice 106 may detect the let go gesture command and a positionassociated with the virtual industrial automation device 806 in thevisualization 114 associated with the AR environment. The head mounteddevice 106 may then un-map the virtual industrial automation device 806from the user's hand and position the virtual industrial automationdevice 806 in the detected position where the user 104 uncurled theuser's fingers. That is, the head mounted device 106 may modify thevisualization 114 associated with the AR environment to display that thevirtual industrial automation device 806 is not mapped to the user'shand. Thereafter, the user 104 may move the user's hand in thevisualization 114 associated with the AR environment and the virtualindustrial automation device 806 may not move with the user's handwithin the visualization 114.

With the foregoing in mind, FIG. 10 illustrates a flow chart of a method1000 for displaying and modifying the visualization 114 based on thegrasping gesture 808 performed by a user 104. Although the followingdescription of the method 1000 is described in a particular order, itshould be noted that the method 1000 is not limited to the depictedorder, and instead, the method 1000 may be performed in any suitableorder. Moreover, although the method 1000 is described as beingperformed by the head mounted device 106, it should be noted that it maybe performed by any suitable computing device communicatively coupled tothe head mounted device 106.

Referring now to FIG. 10, at block 1002, the head mounted device 106 mayreceive image data of the physical space associated with the real-worldenvironment of the user 104. In some embodiments, the head mounteddevice 106 may acquire the image data via one or more cameras 202. Thehead mounted device 106 may then process the acquired image data anddisplay the visualization 114 based on the image data with respect tothe physical space associated with the real-world environment of theuser 104. For example, the cameras 202 may acquire image data ofreal-world objects in the real-world environment surrounding the user.The real-world objects may include physical structures, the user's body,other real-world objects, or portions thereof. At block 1004, the headmounted device 106 may generate and display the visualization 114 basedon the acquired image data and computer-generated content. For example,the head mounted device 106 may display the visualization 114 on adisplay to the user 104 that includes both real-world andcomputer-generated content, such as the one or more virtual industrialautomation devices 806, 808.

At block 1006, the head mounted device 106 may receive a selection of avirtual industrial automation device 804, 806, such as a conveyorsection, based on a gaze gesture command or a palm selection gesture 802command as described above. After the head mounted device 106 receivesthe selection of the virtual industrial automation device 804, 806 basedon the gesture command performed by the user 104, the head mounteddevice 106 may receive image data associated with the gestures or handsof the user 104 at block 1008. In some embodiments, the head mounteddevice 106 may acquire the image data associated with the user 104 viathe one or more cameras 202. The head mounted device 106 may thenanalyze the acquired image data for characteristics associated with thegrasping gesture command 808. If a threshold of one or morecharacteristics for the grasping gesture command match a stored,learned, or otherwise interpretable command, the head mounted device 106may determine a corresponding command to be performed by the headmounted device 106 based on the image data associated with the user 104at block 1010. For example, in response to the determined command, thehead mounted device 106 may determine one or more mapping points betweenthe user's hand that performed the grasping gesture 808 and the selectedvirtual industrial automation device 806. At block 1012, the headmounted device 106 may then modify the visualization associated with theAR environment based on the determine command by mapping the selectedvirtual industrial automation device 806 to the user's hand at the oneor more mapping points in the visualization 114 associated with the ARenvironment. That is, the head mounted device 106 may modify thevisualization in real-time or substantially real-time to position theselected virtual industrial automation device 806 at the one or moreconnection points associated with the user's hand.

Additionally, in some embodiments, the head mounted device 106 maydetect voice commands issued by the user to provide similar interactionsor additional interactions with the virtual industrial automationdevices 804, 806 in the AR environment or with the AR environmentitself. For example, the user 104 may say the voice command “grasp,” or“let go,” “drop,” or “release.” After the head mounted device detectsthe voice command, the head mounted device 106 may perform actions asdescribed herein with respect to the corresponding grasp gesture commandor the corresponding let go gesture command (e.g., release gesturecommand).

After the user 104 has placed a virtual industrial automation device ata position in the visualization 114 associated with the AR environment,the user 104 may wish to move the virtual industrial automation device102 to different locations in the visualization 114 associated with theAR environment. FIG. 11 illustrates a perspective view 1100 of a user104 utilizing the head mounted device 106 to perform a push gesturecommand 1102 or a pull gesture command to move a virtual industrialautomation device 1104 to another position in the visualizationassociated with the AR environment. The head mounted device 106 maydetect the push gesture command 1102 or the pull gesture commandperformed by the user 104 to move (e.g., as an animation) the virtualindustrial automation device 1104 in the visualization associated withthe AR environment. For example, the head mounted device 106 may receiveimage data associated with the user 104 and the user's surroundings.Based on the image data associated with the user and the virtual contentdisplayed in the visualization 114, the head mounted device 106 maydetermine that the gesture command 1102 performed by the user 104 is apush gesture command or a pull gesture command because the user 104 hasplaced both hands on a virtual surface 1104 of a virtual industrialautomation device 1104. The head mounted device 106 may then receivemotion data associated with the user 104 to determine whether thegesture command 1102 is a push gesture command or a pull gesturecommand.

To help illustrate, FIG. 12 is a perspective view 1200 of a user 104utilizing the head mounted device 106 to perform the push gesture 1202to move the virtual industrial automation device 1104 to anotherposition in the forward direction with respect to the user 104 withinthe visualization associated with the AR environment. After the headmounted device 106 has determined that the user's stance is indicativeof either the push gesture or the pull gesture, the head mounted devicemay acquire motion data associated with the user 104. In someembodiments, the motion data may be extracted from the image datareceived by the head mounted device 106. For example, based on the imagedata associated with the user 104, the head mounted device 106 maydetermine a direction and a speed at which the user 104 is moving (e.g.,walking). Based on the direction of movement associated with the user104, the head mounted device 106 may determine that the gesture command1202 is a push gesture command because the user 104 is moving in aforward direction with respect to the position of the user 104 in thevisualization 114 or the gesture command is a pull gesture command ifthe user is moving in a backward direction with respect to the positionof the user 104 in the visualization 114. As illustrated in FIG. 12, thehead mounted device 106 may determine that the gesture command 1202 is apush gesture command because the user is moving in the forward directionwith respect to the position of the user 104. The head mounted device106 may then modify the visualization 114 associated with the ARenvironment to display a movement (e.g., as an animation) of the virtualindustrial automation device 1104 at the same speed as the motion of theuser 104 in the forward direction. In some embodiments, the virtualindustrial automation device 1104 may also be mapped to the hands of theuser 104 in the same manner described above and the movement of thevirtual industrial automation device 1104 may be linked to the movementof the mapped hands.

In some embodiments, the head mounted device 106 may display ananimation of the movement of the virtual industrial automation device1104 after the motion of the user 104 associated with the gesturecommand 1202 is complete. For example, the head mounted device 106 maymodify the visualization 114 to display the animation that the virtualindustrial automation device 1104 is moving after the user 104 hascompleted a pushing or pulling motion associated with the gesturecommand 1202. The head mounted device 106 may determine a virtual forceassociated with the gesture command 1202 performed by the user 104 suchthat the head mounted device 106 may apply the virtual force to thevirtual industrial automation device 1104 in the AR environment tosimulate a movement of the virtual industrial automation device 1104 inthe physical world.

For example, the head mounted device 106 may receive a virtual weightassociated with the virtual industrial automation device 1104 from thedatabase 112. In one embodiment, the virtual weight may be configurableby the user 104. In another embodiment, the virtual weight may be basedon specification data associated with the physical counterpart device tothe virtual industrial automation device 1104.

In any case, the head mounted device 106 may determine an angle that theuser 104 is pushing or pulling the virtual industrial automation device1104. For example, the head mounted device 106 may determine adirectional vector extending from the user's arms or hands and comparethe directional vector to a horizontal axis associated with the virtualindustrial automation device 1104. The head mounted device 106 may thendetermine an angle associated with the pushing or the pulling gesturemotion performed by the user based on the comparison between thedirectional vector and the horizontal axis. Additionally, the headmounted device 106 may determine a speed of the user's hands or arms inthe motion associated with the gesture command 1202 based on motion dataassociated with user 104. The head mounted device 106 may then determinea virtual force based on the determined angle and speed associated withthe user's gesture motion and the virtual weight associated with thevirtual industrial automation device 1104. The head mounted device 106may then apply the virtual force to the virtual industrial automationdevice 1104 after the user 104 has completed the user's gesture motionassociated with the gesture command (e.g., the push gesture command orthe pull gesture command). That is, the head mounted device 106 maydisplay an animation of the virtual industrial automation device 1104moving in the visualization 114 that corresponds to the direction andthe speed associated with the virtual force applied to the virtualindustrial automation device 1104.

In one embodiment, the virtual industrial automation device 1104 mayhave one or more friction parameters associated with the virtualindustrial automation device 1104 in the AR environment. The headmounted device 106 may display an animation of the virtual industrialautomation device 1104 moving slower and/or stopping over time based onthe one or more friction parameters associated with the virtualindustrial automation device 1104 in the AR environment.

In one embodiment, the head mounted device 106 may differentiate betweenthe user 104 performing a push gesture command and a pull gesturecommand by determining an orientation of the user's fingers. Forexample, the head mounted device 106 may detect the push gesture commandby determining that the user 104 has placed both hands on a virtualsurface of a virtual industrial automation device and that the user'sfingers are extended (e.g., straight or upward). In another example, thehead mounted device 106 may detect the pull gesture command bydetermining that the user 104 has placed both hands on a virtual surfaceof a virtual industrial automation device and that the user's fingersare curled around a virtual edge of the virtual industrial automationdevice.

Similar to the push gesture command and pull gesture command of FIGS. 11and 12, in some embodiments, the head mounted device 106 may also detecta nudge gesture command to move a virtual industrial automation device1104 to another position in the visualization 114 associated with the ARenvironment. The user 104 may perform the nudge gesture command to causethe head mounted device 106 to modify the visualization 114 to display afiner movement of the virtual industrial automation device 1104 ascompared to a movement of the virtual industrial automation device 1104as a result of the push gesture command or the pull gesture command.That is, after the head mounted device 106 detects the nudge gesturecommand performed by the user 104, the head mounted device 106 may limitthe movement displayed in the visualization 114 of the virtualindustrial automation device 1104 to less than a threshold virtualdistance, such as the length of the user's arm, the length of the user'shand, or the like. In this way, the head mounted device 106 may moreaccurately display a desired movement of the virtual industrialautomation device 1104 over a shorter virtual distance as compared to amovement of the virtual industrial automation device 1104 as a result ofthe push gesture command or the pull gesture command.

The head mounted device 106 may detect the nudge gesture command bydetermining that the user 104 has placed one or both hands on a virtualsurface of the virtual industrial automation device 1104 based on imagedata of the user received by the head mounted device 106 and virtualcontent displayed in the visualization 114. The head mounted device 106may then receive motion data associated with the user 104 to determine adirection and a speed at which an arm or a hand of the user 104 ismoving in the visualization 114 associated with the AR environment. Thehead mounted device 106 may then modify the visualization 114 associatedwith the AR environment to display a movement (e.g., as an animation) ofthe virtual industrial automation device 1104 at the same speed and inthe same direction as the motion of the user's arm or hand.

In some embodiments, the user 104 may wish to lift the virtualindustrial automation device upwards in the visualization 114 associatedwith the AR environment to reposition the virtual industrial automationdevice or view the underside of the virtual industrial automationdevice. The user 104 may perform a lift gesture command that involvesthe user 104 placing the user's hand on the underside surface (e.g., asurface facing the floor of the AR environment) of a virtual industrialautomation device. In one embodiment, the lift gesture command mayinvolve the user 104 bending the knees of the user 104 to simulate alifting motion of an object upwards. In another embodiment, the liftgesture command may involve the user 104 placing the user's hands on afirst surface of the virtual industrial automation device and curlingthe user's fingers around a virtual edge of the virtual industrialautomation device such that the user's fingers are touching an undersidesurface of the virtual industrial automation device. In any case, thehead mounted device 106 may detect the lift gesture command performed bythe user 104 to move (e.g., as an animation) the virtual industrialautomation device 1104 upward in the visualization 114 associated withthe AR environment. For example, based on image data associated with theuser 104 and the virtual content displayed in the visualization 114, thehead mounted device 106 may determine that the gesture command performedby the user 104 is a lift gesture command because the user 104 hasplaced one or both hands on a virtual underside surface of a virtualindustrial automation device.

The head mounted device 106 may then receive motion data associated withthe user 104. Based on the motion data, the head mounted device 106 maydetermine a direction and an angle of movement associated with theuser's hands, arms, or the like, to move the virtual industrialautomation device in the visualization 114 (e.g., via an animation). Forexample, the head mounted device 106 may determine a directional vectorextending from the user's arms or hands and compare the directionalvector to a vertical axis associated with the virtual industrialautomation device 1104. The head mounted device 106 may then determinean angle associated with the lifting gesture motion performed by theuser 104 based on the comparison between the directional vector and thevertical axis. Additionally, the head mounted device 106 may determine aspeed of the user's hands or arms in the motion associated with the liftgesture command based on motion data associated with user 104. The headmounted device 106 may then move the virtual industrial automationdevice 1104 via the animation in the visualization 114 after the user104 has completed the user's gesture motion associated with the liftgesture command. That is, the head mounted device 106 may display ananimation of the virtual industrial automation device 1104 moving in thevisualization 114 that corresponds to the determined direction and thedetermined speed associated user's lift gesture motion. In someembodiments, the virtual industrial automation device may also be mappedto the hands of the user 104 in the same manner described above and themovement of the virtual industrial automation device 1104 may be linkedto the movement of the mapped hands.

In some embodiments, the head mounted device 106 may be communicativelycoupled to one or more haptic feedback devices (e.g., actuators) thatprovide vibrations to the user 104 based on one or more conditionsdetermined by the head mounted device 106 associated with a movement ofthe user 104, a movement of the virtual industrial automation device, orthe like, in the visualization 114 associated with the AR environment.The haptic feedback devices may be worn, or otherwise attached, to theuser 104 or portions of the user 104, such as the user's hands, fingers,feet, legs, or any other suitable body part. The head mounted device 106may send a signal to the haptic feedback devices to provide vibrationalfeedback to the user 104 to indicate one or more conditions associatedwith a movement of the user 104, a movement of the virtual industrialautomation device, or the like, in the visualization 114 associated withthe AR environment. For example, the head mounted device 106 may send asignal to the haptic feedback devices to provide a vibration to the user104 after detecting that the user 104 has pushed a virtual industrialautomation device into a wall or other boundary. In another example, thehead mounted device 106 may send a signal to the haptic feedback devicesto provide a vibration to the user 104 to notify the user of a messageor an alert. It should be understood that the examples provided aboveare intended to be non-limiting and that the head mounted device 106 maysend a signal to the haptic feedback devices to provide vibrationalfeedback to notify the user of any conditions associated with virtualobjects, the user 104, the AR environment, or the like. In someembodiments, the vibrational feedback provided to the user 104 may alsobe accompanied by voice alerts or notifications to the user 104.

In addition to facilitating the movement of a virtual industrialautomation device by a user 104 to different locations within thevisualization 114 associated with the AR environment, the head mounteddevice 106 may facilitate a rotation of the virtual industrialautomation device along one or more axes of rotations of the virtualindustrial automation device in the visualization associated with the ARenvironment. FIG. 13 illustrates a perspective view of the user 104utilizing the head mounted device 106 to perform a rotate gesturecommand 1302 to rotate a virtual industrial automation device 1304 alonga rotational axis 1306 of the virtual industrial automation device 1304within the visualization 114 associated with the AR environment. Thehead mounted device 106 may receive image data associated with the user104. Based on the image data associated with the user 104 and virtualcontent displayed in the visualization 114 associated with the ARenvironment, the head mounted device 106 may detect the rotate gesturecommand 1302 performed by the user 104 by determining that the user 104has placed one or both hands on a virtual edge of the virtual industrialautomation device 1304. In one embodiment, the head mounted device 106may detect that the user 104 has placed one or both hands on the virtualedge of the virtual industrial automation device 1304 by determiningthat the positions of the user's hands (e.g., fingers, palm, wrist, or acombination thereof) align or overlap with a boundary (e.g., virtualedge) where two virtual surfaces of the virtual industrial automationdevice 1304 intersect. In one embodiment, the head mounted device 106may detect that the user 104 is performing a rotate gesture command(e.g., as compared to a push gesture command, a pull gesture command, ora nudge gesture command) if the user 104 is moving the user's arm or theuser's hand at an angle with respect to the virtual industrialautomation device 102.

Alternatively, the user may issue the voice command “rotate” beforeperforming the gesture or while performing the gesture. The head mounteddevice 106 may then detect the voice command and distinguish theassociated rotate gesture command from the push gesture command or thepull gesture command. In one embodiments, after the head mounted device106 issues the voice command “rotate,” the head mounted device 106 maymodify the visualization to display one or more permissible rotationalaxes (e.g., 1306) that the user 104 may rotate the virtual industrialautomation device 1304 about.

The head mounted device 106 may then receive motion data associated withthe user 104. In some embodiments, the motion data may be extracted fromthe image data associated with the user 104. The head mounted device 106may then determine a direction of rotation and a speed of rotation basedon the movement of the user's hands or arms on the virtual edge of thevirtual industrial automation device 1304. For example, if the user'shand appears to be pushing downwards on the virtual edge of the virtualindustrial automation device 1304, the head mounted device 106 maydetermine that the direction of rotation is counterclockwise withrespect to an axis of rotation of the virtual industrial automationdevice 1304. In another example, if the user's hand appears to bepushing upwards on the virtual edge of the industrial automation device1304, the head mounted device 105 may determine that the direction ofrotation is counter-clockwise with respect to an axis of rotation of thevirtual industrial automation device 1304.

In some embodiments, the head mounted device 106 may receivespecification data associated with the virtual industrial automationdevice 1304 that includes one or more permissible axes of rotation 1306associated with the virtual industrial automation device 1304. In someembodiments, the permissible axes of rotation 1306 may correspond toaxes of rotation associated with the actual counterpart device in thereal world. For example, a permissible axis of rotation 1306 maycorrespond to an axis of rotation 1306 that extends through the centerof mass of the actual counterpart device.

Additionally, the head mounted device 106 may prevent a user 104 fromrotating the virtual industrial automation device 1304 to an orientationthat is not possible in the real world or that would prevent thefunctioning of the actual counterpart device in the real world. Forexample, if the user 104 attempts to rotate a virtual conveyor sectionto an orientation in which a portion of the conveyor section is embeddedin the ground or the conveyor side of the conveyor section is facing theground, the head mounted device 106 may stop the rotation of the virtualindustrial automation device 102 in a position before the virtualindustrial automation device 102 is rotated to the impermissibleposition. In one embodiment, the head mounted device 106 may permit theuser 104 to rotate the virtual industrial automation device 1304 to animpermissible orientation but modify the visualization 114 to display analert or an error notification that the virtual industrial automationdevice is in an impermissible orientation.

After, the head mounted device 106 receives the specification dataassociated with the virtual industrial automation device 1304, the headmounted device 106 may determine the axis of rotation 1306 based on aposition, a motion, or both, of the user's hands on the virtual edge ofthe virtual industrial automation device 1304 and the specificationdata. For example, the head mounted device 106 may determine the axis ofrotation 1306 based on possible rotational directions of rotations thatmay be applied to the virtual industrial automation device 1304 from theposition, the motion, or both, of the user's hands on the virtual edgeof the virtual industrial automation device 1304.

In some embodiments, the user 104 may adjust the axis of rotation 1306.For example, after detecting the rotate gesture command performed by theuser 104, the head mounted device 106 may modify the visualization 114to display a default axis of rotation 1306. The user 104 may then selectthe default axis of rotation by performing an axis repositioning gesturecommand by placing one or both hands on a portion of the default axis ofrotation 1306 to select it or perform a gesture command (e.g., the gazegesture command) to select it. After the head mounted device 106 detectsthe axis repositioning gesture command or the gaze gesture command, thehead mounted device 106 may map the axis to one or more connectionpoints on the hand or hands of the user (e.g., the fingers or the palmof the user). Thereafter, the user 104 may move the user's hand and thehead mounted device 106 may modify the visualization 114 to move (e.g.,as an animation) the axis to the position of the user's hand or hands.In one embodiment, for example, the head mounted device 106 may map theaxis of rotation to the user's hand as if the user was grabbing a pole.The user 104 may then adjust the position or the orientation of the axisto a desired position or a desired orientation.

After the head mounted device 106 determines the direction of rotationand the speed of rotation, the head mounted device 106 may then modifythe visualization 114 to display a rotation of the virtual industrialautomation device 1304 about the determined axis of rotation at thedetermined direction and speed of rotation, as illustrated by FIG. 14.

In some embodiments, the user 104 may rotate and move a virtualindustrial automation device 1304 simultaneously. For example, the headmounted device 106 may detect that the user has performed a push gesturecommand, a pull gesture command, a nudge gesture command, or the like,at an angle. As described above, the head mounted device 106 may displaya rotation of the virtual industrial automation device at the determinedangle about a rotational axis associated with the virtual industrialautomation device 1304 and a movement of the virtual industrialautomation device 1304 at a speed and in a certain direction associatedwith the gesture command.

In some embodiments, the size of the virtual industrial automationdevice 1304 may be too large to conveniently manipulate (e.g., rotate,move, push, or pull) in the visualization 114 associated with the ARenvironment. In such embodiments, the user 104 may be able to perform ascale down command to reduce the size of the virtual industrialautomation device 1304 in the visualization 114. For example, after theuser 104 has selected a virtual industrial automation device (e.g., viaa gaze gesture command), the user may issue a voice command, such as“scale down,” “smaller,” or the like, to reduce the size of the virtualindustrial automation device 1304. Additionally, the user 104 may beable perform a scale up command to increase the size of the virtualindustrial automation device 1304 in the visualization. For example,after the user 104 has selected a virtual industrial automation device(e.g., via a gaze gesture command), the user may issue a voice command,such as “scale up,” “larger,” “grow,” or the like, to increase the sizeof the virtual industrial automation device 1304. In addition, the user104 may scale up the virtual industrial automation device using handmotions. For example, the user 104 may extend his hands at two edges orends of the virtual industrial automation device and move them outward.Alternatively, the user 104 may scale down the virtual industrialautomation device by extending his hands at two edges or ends of thevirtual industrial automation device and moving them inward towards eachother.

With the foregoing in mind, FIG. 15 illustrates a flow chart of a method1500 for displaying and modifying a visualization 114 based on one ormore gesture commands performed by the user 104 to move a virtualindustrial automation device in a visualization 114 associated with anAR environment. Although the following description of the method 1500 isdescribed in a particular order, it should be noted that the method 1500is not limited to the depicted order, and instead, the method 1500 maybe performed in any suitable order. Moreover, although the method 1500is described as being performed by the head mounted device 106, itshould be noted that it may be performed by any suitable computingdevice communicatively coupled to the head mounted device 106.

Referring now to FIG. 15, at block 1502, the head mounted device 106generate and display the visualization 114 based on virtual content andreceived image data associated with the user 104 and the real-worldenvironment of the user 104. For example, the head mounted device 106may display a visualization 114 that includes a virtual industrialautomation device positioned in the real-world environment of the user104 in real-time or substantially real-time.

At block 1504, the head mounted device 106 may receive image dataassociated with a gesture command performed by a user 104. For example,the gesture command may include a push gesture command, a pull gesturecommand, a nudge gesture command, a rotate gesture command, or the like.The head mounted device 106 may then analyze the acquired image data forcharacteristics associated with the one or more gesture commands. If athreshold of one or more characteristics for a particular gesturecommand match a stored, learned, or otherwise interpretable command, thehead mounted device 106 may determine a corresponding command to beperformed by the head mounted device 106 based on the image dataassociated with the user 104 at block 1506. For example, as describedabove, a push gesture command involves the user 104 placing both handson a virtual surface of a virtual industrial automation device. Based onthe image data associated with the gesture performed by the user, thehead mounted device 106 may determine that the gesture commandcorresponds to the push gesture command if the head mounted device 106determines that the both of the user's hands are placed on a virtualsurface of a virtual industrial automation device.

After determining the gesture command, at block 1508, the head mounteddevice 106 may receive motion data associated with the user 104. In someembodiments, the motion data is extracted from received image dataassociated with the user 104. At block 1510, the head mounted device 106may determine one or more motion characteristics associated with thegesture command performed by the user 104. For example, with regard tothe push gesture command, the head mounted device 106 may determine adirection and a speed at which the user 104 is moving (e.g., walking)based on the motion data. It should be noted that the motion datadescribed in the present disclosure may also be acquired using velocitysensors, position sensor, accelerometers, and other suitable speeddetection sensors disposed on the head mounted device 106.

At block 1512, the head mounted device 106 may then modify thevisualization 114 associated with the AR environment based on thedetermined gesture command and motion characteristics. For example, withregard to the push gesture command, the head mounted device 106 maymodify the visualization associated with the AR environment by movingthe virtual industrial automation device at the same speed and in thesame direction as the movement of the user.

Additionally, in some embodiments, the head mounted device 106 maydetect voice commands issued by the user to provide similar interactionsor additional interactions with the virtual industrial automationdevices in the AR environment or with the AR environment itself. Forexample, the user 104 may say the voice command “push,” “pull,”“rotate,” “nudge,” “lift,” or the like. After the head mounted devicedetects the voice command, the head mounted device 106 may performactions as described herein with respect to the corresponding gesturecommand (e.g., the push gesture command, the pull gesture command, thenudge gesture command, the rotate gesture command, the lift gesturecommand).

After the user 104 has moved one or more virtual industrial automationdevices to a desire position, the user 104 may wish to combine the oneor more virtual industrial automation devices to form design of theindustrial system. To help illustrate, FIG. 16 is a perspective view1600 of the user 104 utilizing the head mounted device 106 to couple afirst virtual industrial automation device 1602 and a second virtualindustrial automation device 1604 in the visualization 114 associatedwith an AR environment. In the illustrated embodiment, the user 104 mayhave performed grasping gesture commands using each hand to map thefirst virtual industrial automation device 1602 to the user's right handand the second virtual industrial automation device 1604 to the user'sleft hand.

The head mounted device 106 may detect a snap gesture command performedby the user 104 to couple the first virtual industrial automation device1602 and the second virtual industrial automation device 1604. Forinstance, in the illustrated embodiment, the snap gesture command 802performed by the user 104 may involve the user 104 bringing both handstogether while grasping a respective virtual industrial automationdevice in each hand. The head mounted device 106 may detect the snapgesture command after receiving image data associated with the hands ofthe user 104 and the user's surroundings. Based on the image data of theuser 104 and the virtual data displayed in the visualization 114, thehead mounted device 106 may determine that the user 104 is performingthe snap gesture with the first virtual industrial automation device1602 and the second virtual industrial automation device 1604. Forinstance, if the user's hands are moving towards each other and eachhand includes a virtual industrial automation component or device thatmay interface with each other, the image data that illustrates themovement of the hands with the connectable virtual component may detectthe snap gesture. After detecting the snap gesture performed by the user104, the head mounted device 106 may modify the visualization 114associated with the AR environment to couple (e.g., snap) the firstvirtual industrial automation device 1602 with the second virtualindustrial automation device 1604 at one or more predeterminedconnection points, as shown in FIG. 17. In some embodiments, the headmounted device 106 may provide a snapping motion or a magnetic motionwhen coupling the first virtual industrial automation device 1602 andthe second virtual industrial automation device 1604. That is, the firstvirtual industrial automation device 1602 and the second virtualindustrial automation device 1604 may be brought together at a certainspeed until they are within a threshold distance of each other. At thattime, the first virtual industrial automation device 1602 and the secondvirtual industrial automation device 1604 may accelerate or increase itsspeed toward each other to mimic a magnetic attraction or snap effect.In one embodiment, the head mounted device 106 may provide a snappingsound that may accompany the coupling of the first virtual industrialautomation device 1602 and the second virtual industrial automationdevice 1604 together. The snapping sound may correspond to a pop sound,click sound, or other sound (e.g., ring, chime) that conveys to the user104 that the first virtual industrial automation device 1602 and thesecond virtual industrial automation device 1604 has connected.

In some embodiments, the head mounted device 106 may determine acompatibility between the first virtual industrial automation device1602 and the second virtual industrial automation device 1604 beforemodifying the visualization associated with the AR environment to couplethe devices together. For example, the head mounted device 106 mayreceive compatibility data associated with the first virtual industrialautomation device 1602 and the second virtual industrial automationdevice 1604 from the computing system 110 or other suitable memorycomponent after detecting the snap gesture command performed by the user104. Based on the compatibility data associated with the first virtualindustrial automation device 1602 and the second virtual industrialautomation device 1604, the head mounted device 106 may determine if thefirst virtual industrial automation device 1602 and the second virtualindustrial automation device 1604 are compatible or not compatible(e.g., whether the real-world counterparts would couple together ornot). The compatibility data may be based on specification data relatedto each of the first virtual industrial automation device 1602 and thesecond virtual industrial automation device 1604. The specification datamay detail devices or components that connect to each other, types ofinterconnects, male counterpart component, female counterpartcomponents, and the like.

After determining that the first virtual industrial automation device1602 and the second virtual industrial automation device 1604 are notcompatible, the head mounted device 106 may display an error message inthe visualization 114 notifying the user 104 of the incompatibility. Insome embodiments, the head mounted device may display a recommendationassociated with the compatibility of the first virtual industrialautomation device 1602 and the second virtual industrial automationdevice 1604 with other virtual industrial automation devices.

After the user 104 has coupled one or more virtual industrial automationdevice into a joint virtual industrial automation device, the user 104may wish to separate the one or more virtual industrial automationdevices from each other using a gesture detected by the head mounteddevice 106. To help illustrate, FIG. 17 is a perspective view 1700 ofthe user 104 utilizing the head mounted device 106 to separate a firstvirtual industrial automation device 1602 and a second virtualindustrial automation device 1604 in a visualization 114 associated withan AR environment. In the illustrated embodiment, the user 104 may haveperformed the grasping gesture command using one or both hands to mapthe joint virtual industrial automation device to one hand or bothhands, respectively.

The head mounted device 106 may detect a separate gesture commandperformed by the user 104 to separate the first virtual industrialautomation device 1602 from the second virtual industrial automationdevice 1604. In the illustrated embodiment, the separate gesture command802 performed by the user 104 may involve the user 104 separating theuser's hands while the user 104 is grasping a different section (e.g.,the first virtual industrial automation device 1602 and the secondvirtual industrial automation device 1604) of the joint virtualindustrial automation device. The head mounted device 106 may detect theseparate command after receiving image data associated with the user 104and the user's surroundings. Based on the image data of the user 104 andthe virtual data displayed in the visualization 114, the head mounteddevice 106 may determine that the user 104 is performing the separategesture with the joint virtual industrial automation device. Afterdetecting the separate gesture performed by the user 104, the headmounted device 106 may modify the visualization 114 associated with theAR environment to separate the first virtual industrial automationdevice 1602 from the second virtual industrial automation device 1604 atone or more predetermined disconnection points, as shown in FIG. 16.

In some embodiments, the head mounted device 106 may determine whetherthe user's hands are positioned about a line or a point of severancebetween the first virtual industrial automation device 1602 from thesecond virtual industrial automation device 1604 in the joint virtualindustrial automation device. For example, the head mounted device 106may determine a position of each hand of the user along a joint virtualindustrial automation device. The head mounted device 106 may thendetermine that a line or a point of severance associated with the jointvirtual industrial automation device is located between the positions ofeach user's hands along the joint virtual industrial automation device.In some embodiments, the head mounted device 106 may detect a gazegesture command performed by the user 104 and directed towards the jointvirtual industrial automation device. After detecting the gaze gesturecommand performed by the user 104, the head mounted device 106 maydetermine one or more severance joints in the joint virtual industrialautomation device and modify the visualization to display the one ormore determined severance joints. For example, the head mounted device106 may determine that a joint virtual industrial automation device hasa first severance joint between a first and a second virtual industrialautomation device in the joint virtual industrial automation device, andthe head mounted device 106 may determine that the joint virtualindustrial automation device has a second severance joint between thesecond virtual industrial automation device and a third virtualindustrial automation device. The head mounted device 106 may thenmodify the visualization 114 associated with the AR environment todisplay the first and second severance joints associated with the jointvirtual industrial automation device.

In one embodiment, the user 104 may place the user's hands at desiredpositions about a desired severance joint to select the severance joint.For example, the head mounted device 106 may determine that the user 104has selected a desired severance joint by detecting the positions of theuser's hands, arms, fingers, or any other suitable body part on eitherside of one of the displayed severance joints. In another embodiment,the user 104 may perform a gaze gesture command directed toward one ofthe severance joints to select the desired severance joint. For example,the head mounted device 106 may determine that the user 104 has selecteda desired severance joint by detecting that the user 104 has performedthe gaze gesture command, as described herein, and the target of thegaze gesture command is one of the displayed severance joints.

If the head mounted device 106 determines that the severance joint(e.g., the line or the point of severance) is not between the positionsof each user's hands along the joint virtual industrial automationdevice, the head mounted device 106 may display an error message in thevisualization 114 associated with the determination. In someembodiments, the head mounted device 106 may display a recommendation tothe user 104 in the visualization. For example, the head mounted device106 may highlight the line or point of severance between the firstvirtual industrial automation device 1602 from the second virtualindustrial automation device 1604 in the joint virtual industrialautomation device.

With the foregoing in mind, FIG. 18 illustrates a flow chart of a method1800 for displaying and modifying a visualization 114 based on a snapgesture or a separate gesture command performed by a user 104 in avisualization 114 associated with an AR environment. Although thefollowing description of the method 1800 is described in a particularorder, it should be noted that the method 1800 is not limited to thedepicted order, and instead, the method 1800 may be performed in anysuitable order. Moreover, although the method 1800 is described as beingperformed by the head mounted device 106, it should be noted that it maybe performed by any suitable computing device communicatively coupled tothe head mounted device 106.

Referring now to FIG. 18, at block 1802, the head mounted device 106 maygenerate and display the visualization 114 based on virtual content andreceived image data associated with the user 104 and the real-worldenvironment of the user 104. For example, the head mounted device 106may display a visualization 114 that includes first virtual industrialautomation device and a second industrial automation device positionedin the real-world environment of the user. At block 1804, the headmounted device 106 may receive image data associated with a gesturecommand performed by the user 104. For example, the gesture command mayinclude a snap gesture command, a separate gesture command, or the like.The head mounted device 106 may then analyze the acquired image data forcharacteristics associated with one or more gesture commands. If athreshold of one or more characteristics for a particular gesturecommand match a stored, learned, or otherwise interpretable command, thehead mounted device 106 may determine a corresponding command to beperformed by the head mounted device 106 based on the image dataassociated with the user 104 at block 1806. For example, as describedabove, a snap gesture command involves the user 104 bringing both handstogether while grasping a respective virtual industrial automationdevice in each hand and the separate gesture command involves the userseparating the user's hands while the user 104 is grasping a differentsection (e.g., the first virtual industrial automation device 1602 andthe second virtual industrial automation device 1604) of the jointvirtual industrial automation device.

After determining the gesture command, at block 1808, the head mounteddevice 106 may determine whether the command is valid based on theidentity of the first virtual industrial automation device 1602 and thesecond virtual industrial automation device 1604. As described above,with regard to the snap gesture command, the head mounted device 106 maydetermine a compatibility between the first virtual industrialautomation device 1602 and the second virtual industrial automationdevice 1604. If the head mounted device 106 determines that the firstvirtual industrial automation device 1602 and the second virtualindustrial automation device 1604 are not compatible, the head mounteddevice 106 may display an error message and/or a recommendation at block1810.

If the head mounted device 106 determines that the first virtualindustrial automation device 1602 and the second virtual industrialautomation device 1604 are compatible, the head mounted device 106 mayproceed to block 1812 and modify the visualization 114 associated withthe AR environment to couple (e.g., snap) the first virtual industrialautomation device 1602 with the second virtual industrial automationdevice 1604 at one or more predetermined connection points.

In some embodiments, the head mounted device 106 may join the wireconnections associated with the first industrial automation device 1602and the second industrial automation device 1604 when coupling the firstindustrial automation device 1602 and the second industrial automationdevice 1604. Based on the connected wire connections, the head mounteddevice 106 may perform logic that associates the first industrialautomation device 1602 with the second industrial automation device1604. For example, after the head mounted device 106 detects a separategesture command performed by the user 104 associated with a jointvirtual industrial automation device, the head mounted device 106 maydisplay one or more severance points or joints associated with the jointvirtual industrial automation device in the visualization 114. The oneor more severance points or joints may be associated with locationswhere the user 104 may decouple the wire connections between the firstindustrial automation device 1602 with the second industrial automationdevice 1604. As described above, after the head mounted device 106detects that the user 104 has positioned the user's hands about a properseverance joint, the head mounted device 105 may separate the firstindustrial automation device 1602 from the second industrial automationdevice 1604 and the wire connections associated with each respectivevirtual device.

Similarly, referring back to block 1808 with regard to the separatefeature, the head mounted device 106 may determine whether the commandis valid by determining whether the user's hands are positioned about aline or a point of severance between the first virtual industrialautomation device 1602 from the second virtual industrial automationdevice 1604 in a joint virtual industrial automation device. If the headmounted device 106 determines that the user's hands are position about aline or a point of severance, the head mounted device 106 may modify thevisualization 114 to display an error message and/or recommendation atblock 1810. If the head mounted device 106 determines that the user'shands are positioned about a line or point of severance, the headmounted device 106 may modify the visualization 114 to separate thefirst virtual industrial automation device 1602 from the second virtualindustrial automation device 1604 at one or more predetermineddisconnection points (block 1812).

Additionally, in some embodiments, the head mounted device 106 maydetect voice commands issued by the user to provide similar interactionsor additional interactions with the virtual industrial automationdevices in the AR environment or with the AR environment itself. Forexample, the user 104 may say the voice command “separate,” “snap,” orthe like. After the head mounted device detects the voice command, thehead mounted device 106 may perform actions as described herein withrespect to the corresponding gesture command (e.g., the separate gesturecommand or the snap gesture command).

In some embodiments, the user 104 may wish to design an industrialsystem from a remote location away from the physical location that theindustrial system may be located after assembly. For example, a user maydesign an industrial system from an office or in another country. Insuch embodiments, the head mounted device 106 may provide a dynamicrotation mode that facilitates the design of an industrial system in avirtual environment. To help illustrate, FIG. 19 is an illustration 1900of the user 104 utilizing the head mounted device 106 to navigate avirtual industrial system 1908 in a virtual environment. The user 104may view the virtual industrial system 1908 from in the virtualenvironment without physically moving. That is, the head mounted device106 may detect various gesture commands (e.g., the gesture commands asdescribe herein) or voice commands that may cause the head mounteddevice 106 to modify the visualization of the virtual environment. Forexample, the user 104 may issue navigational voice commands (e.g., “turnleft,” “turn right,” “forward,” or “backward”). The head mounted device106 may detect the navigational voice commands and modify thevisualization of the virtual industrial system to provide the user 104with a view 1902, 1904, 1906 corresponding to the actual view the userwould have if the user performed those actions in a physical industrialsystem. Additionally, the user 104 may interact with the virtualindustrial system 1908, and portions thereof (e.g., various virtualindustrial devices) using other voice commands and gesture commands asdescribed herein. That is, the user 104 may extend his hands to theedges or sides of the virtual industrial system 1908 and move the handsin a manner (e.g., circular motion) that rotates the virtual industrialsystem 1908. By way of example, the user 104 with the view 1902 mayrotate the virtual industrial system 1908 180 degrees to obtain the view1906 without moving from his position in the corresponding physicalspace.

In some embodiments, the user 104 may wish to have a bird's eyeperspective of the design of an industrial system. As such, the headmounted device 106 may provide the user with a scale down command toreduce the size of the virtual system in the visualization. To helpillustrate, FIG. 20 is a perspective view 2000 of a user 104 utilizing ahead mounted device 106 to view a visualization of a virtual industrialautomation device or a virtual industrial system 2002. The user mayissue a voice command, such as “scale down,” “smaller,” or the like, toreduce the size of the virtual industrial automation device or thevirtual industrial system 2002. In some embodiments, the user 104 may beable perform a scale up command to increase the size of the virtualindustrial automation device or the virtual industrial automation device2002 in the visualization. For example, after the user 104 has selecteda virtual industrial automation device (e.g., via a gaze gesturecommand), the user may issue a voice command, such as “scale up,”“larger,” “grow,” or the like, or scale gesture commands described aboveto modify the size of the virtual industrial automation device 1304.

Although certain embodiments as described herein refer to displaying ormodifying a visualization that includes the user's surrounding, virtualobjects, virtual information, or the like, on a display of, for example,the head mounted device 106, it should be understood that, in otherembodiments, the display may be a transparent display allowing the userto see the user's surroundings through the display, and a visualizationthat includes the virtual objects, virtual information, or the like, maybe superimposed on the transparent display to appear to the user as ifthe virtual objects are in the user's surroundings.

Technical effects of the present disclosure include techniques forfacilitating the visualization and the design of an industrial system bya user in an AR environment. The interactive AR system may allow a userto visualize and model various configurations and designs of anindustrial system and the components of the industrial system within thephysical space the industrial system may be located after assembly.Additionally, the interactive AR system provide the user with variousgesture commands and voice commands to interact with virtual objectswithin the AR environment and to navigate the AR environment itselfusing natural hand motions and gestures. In this way, operating in theAR environment may be more easily performed by various operators.

While only certain features of the disclosure have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the disclosure.

What is claimed is:
 1. A system for interacting with virtual objects inan augmented reality environment, comprising: a head mounted device,configured to: receive a first set of image data associated with asurrounding of a user; generate a first visualization comprising aplurality of virtual compartments, wherein each virtual compartment isassociated with one type of virtual industrial automation device,wherein each virtual compartment is represented by a container graphicoverlaid on the first set of image data, and wherein each virtualcompartment comprises a plurality of virtual industrial automationdevices, wherein each virtual industrial automation device is configuredto depict a virtual object within the first set of image data, whereinthe virtual object corresponds to a physical industrial automationdevice; display the first visualization via an electronic display;detect a gesture in a second set of image data comprising thesurrounding of the user and the first visualization, wherein the gestureis indicative of a selection of a respective container graphic thatcorresponds to one of the plurality of virtual compartments; generate asecond visualization comprising a respective plurality of virtualindustrial automation devices associated with the selection, wherein therespective plurality of virtual industrial automation devices shares anindustrial automation device type associated with the selection; anddisplay the second visualization via the electronic display.
 2. Thesystem of claim 1, wherein the gesture comprises the user placing atleast one hand on a virtual surface of the one of the plurality ofvirtual compartments.
 3. The system of claim 1, wherein the gesturecomprises eye movement of the user, wherein the eye movement isassociated with a virtual surface of the one of the plurality of virtualcompartments.
 4. The system of claim 1, wherein the electronic displaycomprises a transparent display.
 5. The system of claim 1, wherein thehead mounted device is configured to: detect a second gesture performedby the user in a third set of image data comprising the secondvisualization, wherein the second gesture is indicative of a selectionof one of the respective plurality of virtual industrial automationdevices; and generate a third visualization comprising an animation ofthe one of the respective plurality of virtual industrial automationdevices moving towards a hand of the user.
 6. The system of claim 5,wherein the second gesture comprises the user extending out at least ahand toward the one of the respective plurality of virtual industrialautomation devices and the user moving the hand from an open palmposition to a u-shaped position.
 7. The system of claim 5, wherein theanimation comprises mapping the one of the respective plurality ofvirtual industrial automation devices to the hand.
 8. The system ofclaim 7, wherein the head mounted device is configured to: track amovement of the hand; and adjust the animation to follow the movement ofthe hand.
 9. The system of claim 8, wherein the animation is configuredto move based on a speed of the movement.
 10. The system of claim 7,wherein the gesture is a first gesture, and wherein the system comprisesdetecting a second gesture in a third set of image data comprising thesurrounding of the user and the second visualization, wherein the secondgesture is indicative of a request to un-map the one of the respectiveplurality of virtual industrial automation devices from the hand. 11.The system of claim 5, wherein the head mounted device is configured to:detect one of the respective plurality of virtual industrial automationdevices moving towards an additional virtual industrial automationdevice; and accelerate the one of the respective plurality of virtualindustrial automation devices towards the additional virtual industrialautomation device in response to the one of the respective plurality ofvirtual industrial automation devices being compatible with theadditional virtual industrial automation device.
 12. The system of claim5, wherein the head mounted device is configured to: detect one of therespective plurality of virtual industrial automation devices movingtowards another virtual industrial automation device; and display anerror message in response to the one of the respective plurality ofvirtual industrial automation devices being incompatible with the othervirtual industrial automation device.
 13. A non-transitory,computer-readable medium comprising computer-executable instructionsthat, when executed, are configured to cause a processor to: receive afirst set of image data associated with a surrounding of a user;generate a first visualization comprising a first virtual industrialautomation device and a second virtual industrial automation device,wherein the first and second virtual industrial automation devices areconfigured to depict first and second respective virtual objects withinthe first set of image data, wherein the first and second respectivevirtual objects correspond to a first and a second physical industrialautomation device; display the first visualization via an electronicdisplay; detect a first gesture in a second set of image data comprisingthe surrounding of the user and the first visualization, wherein thefirst gesture is indicative of a movement of the first virtualindustrial automation device toward the second virtual industrialautomation device; determine a compatibility between the first virtualindustrial automation device and the second virtual industrialautomation device; generate a second visualization comprising ananimation of the first virtual industrial automation device coupling tothe second virtual industrial automation device to create a jointvirtual industrial automation device in response to determining that thefirst virtual industrial automation device and the second virtualindustrial automation device are compatible, wherein the animationcomprises the first virtual industrial automation device acceleratingtoward the second virtual industrial automation device, the secondvirtual industrial automation device accelerating toward the firstvirtual industrial automation device, or both; generate a thirdvisualization comprising an error notification in response todetermining that the first virtual industrial automation device and thesecond virtual industrial automation device are incompatible; anddisplay the second visualization or the third visualization via theelectronic display.
 14. The non-transitory, computer-readable medium ofclaim 13, comprising computer-executable instructions that, whenexecuted, are configured to cause the processor to: display the secondvisualization; detect a second gesture in a third set of image datacomprising the joint virtual industrial automation device and at least aportion of the user, wherein the second gesture is indicative of arequest to separate the joint virtual industrial automation device;generate a fourth visualization comprising a second animation of thejoint virtual industrial automation device separating into the firstvirtual industrial automation device and the second virtual industrialautomation device; and display the fourth visualization via theelectronic display.
 15. The non-transitory, computer-readable medium ofclaim 14, comprising computer-executable instructions that, whenexecuted, are configured to cause the processor to: detect a thirdgesture in a fourth set of image data comprising the joint virtualindustrial automation device, before detecting the second gesture,wherein the third gesture is indicative of a request to display one ormore severance joints associated with the joint virtual industrialautomation device; generate a fifth visualization after detecting thethird gesture, wherein the fifth visualization comprises at least theone or more severance joints and the joint virtual industrial automationdevice; and detect a fourth gesture in a fifth set of image datacomprising at least the one or more severance joints and the jointvirtual automation device, before detecting the second gesture and afterdetecting the third gesture, wherein the fourth gesture is indicative ofa selection of one of the severance joints.